Culturally Insensitive ?? or Innocent Comment ??

So... is a statement culturally insensitive if it's true regarding a specific person and not intended to be insulting in any way?

My example from a few minutes ago. We're talking about college professors that we had from other countries. Everyone has these stories, right? So I said something about how the worst one I had was "right off the boat". Everyone got quiet. An uncomfortable quiet. So I explained. This particular instructor had told us in his broken English that he had arrived by boat to the United States one week before he started teaching. He was, literally, right off the boat. I honestly did not realize there was anything wrong with that phrase (is there?) until the uncomfortable quietness settled over the group. If I had said he was "right off the plane" would that have been okay? If I simply said he had "arrived recently" would that have been better?

Another example, involving FireMan. In fact, he almost got fired for it. The city got sued over FireMan's comment.
He was almost done with the probationary period as a firefighter. The recruits were discussing what to get for dinner that day. FireMan asked the black recruit (yes, there was only one), let's call him "Matt",  if he wanted them to order fried chicken. Now, what you need to understand, is that the day before, they had had a conversation about everyone's favorite foods. Matt had stated that fried chicken was his favorite food. He had also complained that he never got to pick the restaurant when they ate.
So when the discussion of what to eat came up, my husband innocently asked if Matt wanted them to order fried chicken, because Matt had made it known that that was his favorite food. Matt's response? He left in a hurry, obviously angry, and went straight to the Chief's office and reported that FireMan had made a racist comment. He also later sued the city for racism.
Over a year of litigation (and a lot of stress) later, Matt lost the lawsuit, and FireMan was found to be innocent in his statement.

But... do you see where I'm going with this?

I guess I'm still trying to figure this stuff out. It's an ongoing learning process. I mean, I do think we do need to have a certain level of sensitivity to others' feelings, and definitely should never say anything that is intentionally insulting (or could be construed that way). But then again, just about any statement could be construed as however the listener wants to construe it.

Maybe it's a case-by-case thing. Something said in one setting might be perfectly innocent, but in another might not be. I don't know. Just rambling thru my thoughts on the subject