Notrenoir, your dilemma is very much a common one. There are many barriers to "crossing the line", and the barriers grow in number as you approach your middle years, having made huge investments and commitments toward a way of life.
You are only a couple years younger than I. What I've discovered, is that many of the barriers to transitioning a little or a lot, fall away at our age. Having graduated from parenthood, retired from the labor force, and having survived the forces that push people into divorce, it finally comes down to convincing oneself that it's now or never.
If having breasts is more of a sexual fetish than a true gender identity issue, I would not recommend experimenting with breast growth. The effects of PM on your body (breast growth in particular) and your mind (loss of interest in sex, possible anxiety or depression) only tend to complicate your life.
Gender identity dysphoria is a completely different reason to pursue hormone rebalancing whether using pharma or phyto estrogens. Years of mental discomfort can be relieved with appropriate therapeutic counsel and support structures (like this forum).
It's always a personal decision to cross the line or not. Most come to make the right decision in the end, so you're likely to do so, too. Good luck.
Hugs,
Clara