I oscilate back and forth...one part of me says, "The word is just a word. It doesn't matter what word you use, if you're bigotted it'll come through and if you're not, it'll be obvious so who cares." Another part of me sees the view that getting lazy with language and pretending that language doesn't matter makes it hard for people to distinguish appropriate from inappropriate attitudes.
Douglas Hofstadter wrote a rather provocative column 20+ years ago called
A Person Paper on Purity in Language. He wrote it as an aside in the ongoing (and in 1982, very topical) discussion of gender non-neutrality in the English language. He invented a world in which the English IS gender neutral, but, NOT race-neutral, and then proceded to make the same arguments defending this status-quo as were being made to defend gender non-neutrality. It may not prove anything at all, but it certainly makes one think.