While it is true that a language mirrors a society's worldview, the idea of someone going pale, does not mean one becomes white. It simply means one's natural colour becomes lighter and duller as there is less blood irrigating the skin. A healthy person's natural colour can go from very light to very dark. If one becomes pale, then the normal tone becomes lighter and develops a different tone: it can become whiter, yellowish, or greyish.
If I read that a black character became pale or their colour drained from their face, I picture them with a lighter but duller (sickly in the sense they look sick) tone than how I usually picture them.
In this particular example, I suggest you may be looking at the expression a bit too literally. Allow me to give a different example.
The colour 'orange' derives its name from the colour of the fruit, orange (in the past it would have been called reddish yellow or similar). The term 'olive skin' also derives from a fruit: it describes a skin tone similar to the tone of the olive fruit. Or better yet, similiar to tone of a variety of olive. I live in Portugal, and I have never seen an olive with a colour similar to my skin or most people I know. They're mostly black or green, sometimes greenish-yellow and sometimes dark reddish. I do know the variety whose colour is similar to the particular skin tone being described, but reading that someone has olive skin will always wrench me out of the story because the worl olive automatically ellicits the colours black or green.
In this particular example, the English language associates a colour to a thing which, in my culture, has a radically different colour.
I realize that I am forced to describe [...] from an English speaker's POV.
Not necessarily. When I describe someone's skin tone I cannot bring myself to describe it as olive because it's so hard for me to associate the word to the correct colour. Therefore, I use other alternatives.
In your particular case - how to describe paleness in a dark skinned person - I suggest you look up images of people with the skin tone you're after and see if you can find any of sick people. Some medical papers have photos of patients (typically with their eyes blackened out for privacy reasons - I often read such papers for research and I'll warn you that, depending on the disease being studied/described, some of those images may be upsetting) which may help you see the colour. Then, once you know what it looks like, come up with a descriptor that fits it.
You may want to look up words used to describe and distinguish colours (look up 101 texts for art students) to understand what 'becoming pale' means in terms of colour change (personally, I'd say it loses saturation, though that's a bit too technical to describe people).