I wouldn't put much thought behind it. Like it was mentioned earlier, because we have a fixation of our own chests, we become more aware of it than we normally would. Most people either don't notice or don't care.
Back in high school I did a bit of a social experiment where I filled a couple of balloons with a mixture of shaving cream and water (which causes the mixture to foam, so it's not too heavy but still volumizing) and wore them with a C cup bra for a week. The vast majority of people at my school didn't notice, not even the teachers. When I finally stopped wearing them, the only ones who said anything were my friends who naturally would notice since we interacted on a regular basis.
If you're worried about acceptance from a future friend or possibly SO, don't be. I don't know your orientation, but assuming you're straight or bi with a preference to females, there's a lot of women out there who either don't care or actually enjoy breasts. My current *cough* romantic interest is really big into androgyny and LOVES breasts, regardless if they're on a guy or not. Prior to her, my ex would often times play with my chest (which I had grown out with HRT a couple years prior), and was saddened when I had my breasts reduced (story for another time).
All in all, the noticeability of your chest really depends on how you go about presenting yourself. Clothing for men often times aren't designed for showing contour and shapely figures, and often times hides your bodily figure. If you're self conscious about your chest then a decent sports bra and loose fitting t-shirt should be enough to hide your chest. I really wouldn't worry about your breasts garnering much attention unless you're in the DD range or bigger