Well, there are many ways to see the way you feel.
Some men experience dysphoria when their body is not "manly enough", for example when they grow breasts as a result of medication. It's a "male-to-male" gender dysphoria. There was a gender therapist/psychiatrist (I think) who accidentally injected himself with a higher-than-intended dose of Estrogen and he said that the potential consequences gave him genderdysphoria as he didn't want to part from being a man with a man's shape.
Another interesting type of dysphoria is male to non-binary. (or FtNB from the other side). It is something that gender therapists are slowly taking on board and sometimes they help people with HRT to get closer to an androgynous appearance.
I am not sure if any of the above reflects what you described but, the point is, dysphoria means a difference between ourselves and our perceived selves. I think that if one accepts to be somehow on the NB scale, acceptance of our bodies becomes easier, and also acceptance of our need to change our bodies "without being fully trans in the most conventional sense" becomes easier.
That said, it is a lie that men don't have breasts. It should be fairly clear now that gynecomastia is a thing. Wikipedia says that 35% of men are affected by gynecomastia at some point in their life. That's one third of the male population, which I think is a wide-enough pool of people to make it "normal". There are more men with gynecomastia, with boobs, out there than blonde men.
Is is unusual to want more boobs? To want more erogenous nipples? Maybe. Biology tells us that gynecomastia comes with age, something that tells the onlookers that the person is not in his prime anymore. So, if humans were just supposed to live for reproduction then, yes, wanting boobs would be really weird. But we now live almost twice as long as we need for reproduction, so we should be at ease with the idea that in the second part of our lives we should be able to do whatever we want and whatever we like with our body. (Our whole life, tbh, but I am seeing it from a conservative point of view).