There is a lot of, often erotic, fiction that spends quite some time describing the sensual qualities of the surface appearance of people (color, shape, texture, smell, sound). Usually that literature chooses color terms that both describe the color well (we all have a clear image of what chestnut hair looks like) and evoke a pleasant sensual image (cockroach hair my be the same color, but most of us wouldn't be attracted by it).
For example, a person might be described as having a cream colored skin, and cream is something that we associate with cake or rich soup or the already-sexualized coffee or other tasty and pleasant foods, so a cream colored skin is loaded with what those foods mean for us: sensuality, delight, plesasure.
I don't see how food analogies must be offensive, except to those to whom any taking note of skin color is offensive. I think that being momentarily overwhelmed by the sensuality of someone's appearance and ignoring for the time being that person's intelligence and wit is completely fine and not a sign of disrespect or shallowness. Your description of your character's skin will be embedded in a depiction of that character's humanity, and if that well-rounded character has "beautiful chocolate skin" in the eyes of another well-rounded character, who both have a multi-faceted relationship with each other, then why not?
The politically correct alternative would not even be to use Pantone color numbers or RGB codes, but to not to refer to skin color at all. But since we cannot avoid perceiving skin color in real life and are always influenced by that perception, showing that influnce on your characters is to me necessary and a sign of xxx.
If your character is a racist, you must show how he perceives skin color in a prejudiced way, maybe by using negative analogies, and if he is in love with the person he looks at, you must show how he perceives skin color in a prejudiced way also, except that here the prejudice is a positive one. Your descriptions are part of your characterization – only not of the person you describe, but of the person who is described as looking at them. So think about who is looking, what their relationship is to the person they look at, and how this person would perceive that other. Then chose the right words to illustrate that perception. And if the right word is "olive" or, an example from a comedian, "whore skin umber", then use that word to effect in your readers those emotions you aim for.
If it is your narrator who describes a character, and not another character, think of the narrator as a character and different from yourself. Give your narrator a personality that is expressed in the way he or she narrates, and choose the decriptive terms to illustrate that personality.