oK then I just have to ask...
Would men's sperm be a significant source of aromatase?

I think so, the study concluded that Leydig cells, germ cells, and seminiferous tubules all showed positive stains for aromatase.