Non-native English speakers wear stars in English-speaking countries, to identify them. We Brits started since we originated the language, then all of our ex-colonies caught on lol..
In seriousness, it wouldn't bother me. If somebody comes to London and doesn't speak English well, so what? Nobody is all learned, are they? I only know a few French phrases (je parle en peu de francais, n'est ce pas?) or some basic German words (Enschulidgung?), but it doesn't mean I'd be daunted visiting Paris or Hamburg.