As I see it, there are a couple of flaws in your reasoning. Blouses frequently have what I guess is called a gusset at the bust line which actually calls attention to the breasts and is a feature usually not seen in men's shirts. Furthermore, and maybe I'm the only one who notices this stuff because I occasionally crossdress, women's clothing buttons on the opposite side of men's clothing, and I would notice this. In fact, years ago, I noticed on VH-1 Behind the Music that Anthony Kiedis was wearing a female coat or top when they profiled the Red Hot Chili Peppers.
Even women's T-shirts have smaller collars, and I would notice that, too, if I saw a guy wearing one. If a guy is going to wear a woman's blouse, he may as well just wear a bra and get it over with. It isn't subtle in my opinion.