No, it is not inherently racist. But, no, you cannot stop some readers from deciding to call it racist.
White has often been associated with cleanliness and purity, for several reasons. These include the Sun & fire (the easiest way to sterilise things and make them safe for humans), which both give out light, and dirt & mud being dark (so a light surface is easy to see as "clean", or dirt-free)
Much of the association of "white" with "good" comes from this association with an item being clean or pure - including, for example, in Japan, or the Native American Hopi tribe. This demonstrates that the symbolism can evolve even in non-white ethnicities.
(As an interesting contrast, the Hambukushu tribe from Zambia had an old legend wherein their sun-god, Nyambi, is said to have passed all people through a fire after creation, to purify them. The myth continues that white-skinned people are those who rushed through the fire, while dark-skinned people are those who passed through slowly - thus, they had a culture where fire brings purification, and white items are pure/good, but white people are not)
When white is taken as a symbol of "good", then black is commonly taken as a symbol of "bad", due to being an obvious opposite.
But, to be perfectly honest - if someone sees "good object is white, bad object is bad", and immediately jumps to "that's racist" with no further indication of that being the case, then they might need to examine their own bias and prejudice. Which means your final point - making black "good" and white "bad" - is, if done purely for these reasons, not merely just as racist, but also attempting to dress it up in virtue signalling.
Put in another context, it's like interviewing a male and a female for a job, deciding that the female would not be able to do the job because she's female (which is sexist), and then deciding to give her the job for which you think she isn't qualified because she's female (which is, itself, sexist), all in an attempt to hide how sexist you're being.
(Doing so for other reason is fine - for example, associating black, void and darkness with emptiness and tranquillity and associating white, fire and the Sun with damage and chaos, or - in the interview context - deciding that while the female currently has far less experience in the field, she has advanced further in a shorter period of time than the male, so has a higher potential to become the better candidate in the long-term with the appropriate training. But, again, you still can't stop people reading additional overtones into your work/decisions.)