Here's a really stupid argument:
1. Term X can be used in a racist way.
2. Other uses of Term X are therefore racist.
It's got to be one of the poorest excuses to call someone a racist I've ever seen. Yet people insist on doing it to unsuspecting politicians or other public figures. It's for this reason that Governor Mitt Romney of Massachussetts has been bamboozled into apologizing for an action that is in no way wrong. Tony Snow has also been criticized for using the same expression in its original, non-racial sense. A tar baby is generally a sticky situation, and nothing about race is implied by this use of the term. It's origins come from an African folk tale, and its function in accounts of sticky situations has continued undisturbed by those who ignorantly coopted it for racist purposes. In the northeast, where Romney is governor, most people have probably never even heard of the racist use of the expression, and those who do encounter it might easily forget it as so far out of their vocabulary that it doesn't enter long-term memory.
Another example I've encountered (in this case only very recently) is "call a spade a spade", which simply means to identify something for what it is. Some racists in the South have apparently called black people spades as a derogatory term. Since I've never hung out with those people, it never would have occurred to me that someone would do so. Why should an uncommon use of a term in a localized region, a use I've never even heard of, make my use of a perfectly normal idiom somehow immoral? Those who treat such statements as racist seem to me to be linguistically unaware at best and incapable of moral reasoning at worst.