Geoffrey Pullum at Language Log has an interesting post about his dislike of the term 'people of color'. I've never been taken with the term myself, but I don't think there are any very strong objections to it that don't also apply to other terms that I readily use. For example, it seems funny to act as if white people have no color, but we do speak that way when we call white people white and non-white people non-white. If it's bad to speak of people of color, then it's bad to speak of people who are non-white. In terms of economy of words, it's awkward to say "people of color" as opposed to "colored people", but the former doesn't have the negative connotations now usually associated with the latter, and if grammatical sleight-of-hand allows for a good result without changing the actual terms much is that so bad? There's no really strong linguistic or moral reason against it. So why not?
Well, Pullum just doesn't like the term. That's it. He doesn't judge anyone as linguistically or morally on the wrong side for using it. He just doesn't like it and doesn't use it himself. I'm not sure I dislike it as strongly as he does, but I've never been especially excited about it, and it has nothing to do with the reasons he gives in his slightly unfair characterization of conservative views on race. (I say slightly unfair, because I think what he's describing does happen, but he doesn't seem to allow for people who have views very similar to his on the moral questions but different views on the political ones, which is exactly where I stand.)
I first encountered the term during orientation in my first year of college, and it struck me as very strange. I can't say I've taken to it more over the years, but I don't have any reason I can think of why I shouldn't like it except ones that rely on bad arguments. I suppose it's better than 'non-white' in one way, because it's not defining people in terms of what they aren't.
On the other hand, I'm not sure I can think of a way that 'people of color' makes any sense to refer exactly to the people it refers to except in the sense that they aren't white. So maybe I just find it deceptive as a way to avoid saying something else that might offend some people. In other words, maybe it avoids overt offense by relying on offensive assumptions that aren't immediately discernible without reflection. But I think this might take more argument than I'm prepared to give at this point.