Yeah, it is problematic since in a lot of Western democracies doing something like that, especially when many people are uncomfortable with it and/or have not given you permission to behave in that way, legally constitutes sexual harassment.
And, as a general moral principle, you should probably stop doing something that you yourself have concluded makes a great deal of people uncomfortable. It doesn't speak very highly of your concern for the comfort of others.
Personally, I grew up in a Puritan small town so I use urinals when there are dividers and otherwise stick to stalls. This is an important point, too, since you may want to recognize that there are also different cultural perspectives on this and you may well be offending people culturally as well as personally.