It's not so much the breast as the whole body. Fingers and toes go through the same thing- smaller in cold, larger in heat.
Basically it's the body (well, skin and all that) expanding outward to cool itself off in heat, and contracting inward to stay warm. I'm female but the same thing happens to me.
Of course, heat and sun are preferable to cold