Perhaps the person who told you this is concerned about how it affects your safety. If you have reasons to worry, then you should not go for it. As a Muslimah, you ought to prefer your safety over such an unwanted display of your identity.
Furthermore, it also saves you money which you can spend on greater priorities. Think of the millions who cannot afford a morsel of food or a clean glass of water.
If you are conscientious, you should spend the amount you save in charity. By doing so, you stand to gain immense spiritual blessings in both worlds - provided you do it to please Allah.
I pray to Allah to inspire us to act rightly.