In a perfect world, this wouldn't be an issue, but unfortunately there's a long history of characters --in both realistic and fantasy-based settings --being "color-coded" according to a "lighter skinned = better, more moral, virtuous, good" and "darker skinned = worse, more immoral, evil" system. It's even encoded in our language: "Fair = beautiful" and "dark = sinister." Even more unfortunately, these portrayals, even in fiction, do have real-world impacts --black people are often perceived as threatening and scary even when doing perfectly innocuous activities, sometimes with tragic consequences.
Keep in mind, this is your world, and you control its aspects. Since you're aware of this as a problem, why not mitigate it? Red, orange and gold are alternate colors that could be associated with the sun. Conversely, you could make sure there was more than one dark-skinned group in your book, or that some of the heroic sun elves were known to have survived.
I applaud you for being cognizant of this, and hope that you will address it. As a black person who is a long-time fan of fantasy, it's a real turn-off for me to see the old racist tropes persisting even in newer literature, from authors who really ought to know better.