You're right, check this out:

In most people, when sugar in any form is consumed, the pancreas releases insulin. Breast tissue, for example, contains insulin receptors, and insulin is a powerful stimulant of cell growth. One group of Australian researchers concluded that high levels of insulin and insulin-like growth factor (IGF) may actually be causative of cancers of the breast, prostate, endometrium and pancreas. A broad study conducted in 21 countries in Europe, North America and Asia concluded that sugar intake is a strong risk factor contributing to higher breast cancer rates, particularly in older women. A four-year study at the National Institute of Public Health and Environmental Protection in the Netherlands compared 111 biliary tract cancer patients with 480 healthy controls. Sugar intake was associated with more than double the cancer risk.