I'm going to tell you a true story, and then I'll let you come to your own conclusions about it.
A little while back, I stopped at a rest stop on the interstate because I had to use the bathroom. I went into the men's room, as I usually do, because that's what I was taught to do, having been raised without conflict as a boy. While standing at the sink washing my hands, a man came into the restroom. He took one look at me, and quickly turned around and walked out, thinking he had stepped into the ladies' room by mistake. This is not an unusual occurrence, and is not the first time this has happened to me.
Well, I recently stopped off at the very same rest stop, but this time, I decided to go into the ladies' room instead. According to the social rules I was taught, this is an enormous violation. But the rules weren't designed with transgender individuals in mind. Well, I was standing at the sink washing my hands (in a weird reverse image reflection of the earlier time in the men's room), and a woman came into the restroom, leading her school-aged daughter by the hand. They walked right past me without flinching.
Now I've got all the physical plumbing of a man, but not only do I feel more comfortable in the ladies' room than I do in the men's room, but men and women seem to feel more comfortable with it, too.
As I said, you can draw your own conclusions from this. I was very convincingly dressed and groomed as a woman the day I went into the ladies' room, whereas I wasn't trying especially hard to be feminine the day I used the men's room. Is biology the more important factor in determining proper restroom usage, or is psychology more important? Is it entirely a superficial issue - whether other people would guess you are male or female just to look at you - and if so, why are we gambling people's comfort and safety on something as unsubstantial as image?
If I go into the men's room looking like a girl, I fear for my own safety. But if I go into the ladies' room and am not convincing enough as a girl, I risk threatening other people's safety. Yet I know I wouldn't hurt anyone, and I'm not there for the purpose of violating anyone's privacy. Other people may not know that, but is it my responsibility to put myself at real risk of harm, in order to avoid imposing on anyone else the false threat of harm? Is that the price transgender individuals have to pay for being different?