[PleasantlyFascinated]

I guess its going to be different for each person, but is it fair to say that I can expect heavily reduced muscle growth if I workout while on PM?

I would definitely expect stalled muscle growth.  The phytoestrogens will bind to androgen receptor sites that you would want available for binding T and DHT, for muscle hypertrophy.  I would expect to have to double down on hypertrophy stimulus, just to maintain what you have.