Hmmm.  I know quite a few very fit women, but it is also no secret that women in general can't compete on an athletic basis with men.  Testosterone does a lot to increase muscle mass, and decreasing testosterone will certainly result in a significant loss of that muscle mass.  If you have less muscle what you do have has to work harder and will tire more quickly.
I used to push my body very hard, especially when I was in the service.  I've stopped most of that (because of injury) but I'm also sure that actively feminizing myself will accelerate the muscle loss.
I don't know about the brain re-wiring causing a lack of stamina.  I haven't encountered that yet but then I still can't push myself very hard.  I have always heard, however, that many women could actually tolerate pain and discomfort better than men, to prepare then better for child-birth.  Still, time will tell!
Don't discount the getting older part also.  It sucks!  
