It specifically describes a skin colour and it is intended as a slur. These are indisputable facts.
I disagree that Dickens used it in exactly the same way as it is used now. Not least because England is a very different place now. It is one of the most ethnically diverse countries in the world but, in Dickens’ time, it was overwhelmingly white. Therefore, the notion that “gammon” could have been construed as having racial overtones would have been absurd back then. Not so now. The fact that we are even having this discussion confirms as much.
There are all manner of words and expressions that were acceptable in the past - or even that had a subtly different meaning in the past - that are no longer considered acceptable. Among them are myriad words and phrases related to skin colour. Why should “gammon” not be included just because it refers to white people?
It doesn't describe skin colour (maybe you should go to see your doctor), it describes a mindset and attitude. You could call a non-white person a gammon and everyone would know exactly it would mean.
Dickens used it to describe a way of thinking and that is exactly how it's used now.
If people don't like being called a gammon, then they can stop being a gammon. It's that easy.