Not at all, quite the contrary in fact.
And yet the words would be exactly the same, just spoken by different people and the reaction would be completely different.
As someone else has said, if people interpret the chant as racist, then at least in some small way it must be, but there are more than one way to interpret something.
Maybe they didn't take offense because they decided the people who were calling them chocolate man didn't mean to offend and also maybe took into account the cultural situation. The fact that there is a high likely hood that they have never come in contact with a black person before.