[CM213]

From my research it is the lack of testosterone in ones body that causes the atrophy of ones penis. Just like if a woman takes testosterone it will enlarge her clitoris. Now obviously hrt is a combination of introducing female hormones and trying to stop, or eliminate male hormones from be produced or being able to be used by the body. So it does work hand in hand and even faster when hrt is also involved. 

But if someone had their testicles removed and did not use testosterone replacement nor did they start HRT their penis would still shrink due to the lack of testosterone in their system.