The basis of shari’ah ruling is the principle of generality. This means, what is naturally beneficial to an individual is permissible and what is naturally harmful to an individual is prohibited. What is known is that when a woman goes out wearing a perfume, she will be a cause of temptation. Therefore, it is forbidden for her to use the perfume despite the fact that some men may not be even attracted to her perfume.
On the flip side, when men wear perfume, what is naturally known is that It does not cause any temptation as it does when women wear it. However, if there are specific fragrances that cause temptation, such perfumes become prohibited to be used. Not as a general principle that perfume is forbidden for men, but as a fact that whatever causes temptation and leads to discord is prohibited for both men and women. Shariah does not consider anomalies (exceptional cases), but looks at general courses regarding men and women. That does not affect the fact that matters are by their intents and reasons.
Everything that has the potential to cause temptation should be discouraged, such as a woman staring at a man and vice versa. This may cause problems for both genders, thus it is forbidden for them, even though, there are special cases whereby staring at the opposite gender does not affect some.