I think we should abolish the notion that sexual attraction is something that occurs between genders, and reframe it as something that occurs between people.
Yes, it's probably true that a great number of people generally have a preference for one gender or the other (and it's still okay to acknowledge this), but even a man exclusively attracted to women is not going to be attracted to every female alive. Gender is just one of many attributes he is attracted to, even if it is a particularly strong one.
But the idea of attraction occurring between genders puts us into a framework where all women are expected to be attractive to all men - usually by a communally pooled set of standards - where I argue the human species would be better served by considering attraction a phenomenon (not necessarily mutual) that occurs between specific people.
If you're a woman, you can't possibly expect all men to be attracted to you; and this way, if one man says you're ugly, it's more obvious that this is his opinion and not your failing - unless this is a person whose opinion actually matters to you. Really, I think we'd all be much better off this way.
Plus, it would be a lot more considerate (and less assuming) of non-hetero sexualities.