Here's the thing about race: different people have very different ideas about what race means, and how race affects individual's lives.
Some people's ideas of race is inextricably tied with their experience of racism. Some people's idea is that race doesn't really matter, or that it shouldn't really matter. Other people have a basic assumption of which race is the "normal" one, the default, and anything else is (in some way or another) special. And so on.
When you're writing a story or constructing a world, then you are the arbiter of how race really works there. The world works the way you say it does. And, if you don't describe how race works -- people are going to just assume some kind of default. Not addressing race at all is like not addressing whether or not the planet has gravity: You can go with the default, but that doesn't mean you haven't made a decision.
Now, in your case, you have a fairly clear idea of how race works: race exists, but the world is effectively colorblind. Nobody minds, nobody cares, nobody makes anything of it.
My question is this: How do you tell the reader that this is the case? Is the reader envisioning the same multiracial landscape that you are, or are they assuming something very different?
If you never address it at all, I think most readers will assume you just haven't given it any thought. They will assume everybody is "the same," because you haven't said otherwise. It sounds to me like that isn't the result you want.
But. Cluing the reader in to your setting's diversity is not the same as tagging one specific character as being "the black one". If you truly want to spend the effort on creating a world where skin color isn't an issue, you can absolutely do that -- you just need to do the work of actually portraying the world as you imagine it. To portray the world as one with lots of races and no racism, without singling out individual characters as "the ones with race."
You'll have to tailor this to your own story and your own vision of your setting. But some suggestions:
- You can specify early on that the group aren't all white, without saying which of them is black! This in itself might be a testament to people not even caring who's what.
- You can reflect the cultural norm in other characters and people. If the King is black and the Queen is white and nobody ever mentions this as unusual, that's a meaningful statement. If there's a fringe religion where different races are considered to have different "essences" and be "lucky" for different things, and they're generally considered really weird, that's a meaningful statement.
- Give a lot of thought to how and why the world is so non-racist. Is there no local majority of one particular race? Did the different races not originate in different geographical areas, seeing themselves as "normal" and others as "outsiders"? How has race never become a meaningful distinction -- or, how has using it as a distinction become taboo? Answers to those will affect your world in ways you can portray!
You can find many more ideas in similar vein -- figuring out how to get across to the reader that racism just isn't a thing.
All that being said, this can be a very difficult path to walk down if you're aiming for serious worldbuilding. Plainly speaking -- in the real world, race is part of who a person is. It's very difficult to build an authentic world where race exists but has no discernible effect whatsoever. It's intertwined with culture, community, immigration, history, and tradition. If you try to just eliminate all those, you're much less likely to get across "there are black people and white people getting along just fine," and much more likely to be read as "everybody here is white, or feels white."
In that sense, just having specific characters who are POC can be a much easier route to showing how POC fare in the world. I can see why you're thinking of avoiding that, but the other way might not be what you want either.