I don’t know if he could possibly just say something along the lines
of “I just don’t feel that way about people.” If so, how would I work
this into a natural conversation?
Show, don't Tell
How does this character's orientation effect the story? Are they pressured into a sexual or romantic situation? Why would they need to say this to another person?
If it's not part of the story, treat it like the 90% of the iceberg that is hidden under water, it's "there" buoying him up but we don't see it directly.
If it is part of the story, look for a way to present it through a series of actions or reveals, rather than the character just saying so. A character blurting out "I'm not interested in anyone romantically…" early in a novel could read as a Chekov's gun, signaling this is a "problem" he will need to solve by the end. We tend to expect a payoff when characters tell us stuff too easily.
It's also a little weird, which is the point of this question – it doesn't really sound natural. Why explain this to anyone? If he has a romantic obligation to them, like a spouse, I'd expect some narrative build up, soul searching, guilt, something. If it's all settled for him and he feels confident about it, then he would be a little older and wiser and experienced – again I'm wondering why he'd need to put it into words, who does he need to understand this truth about himself?
exposition from a sexpert
If you want a scene that wraps this all up on-the-nose, send him to a brothel with his mates. While they're all upstairs banging he can be drinking and singing songs with the brothel's procuress, an older refined woman. Two new girls can ask about the gentlemen's needs and the older experienced woman shoos them away with a curt explanation from authority.
It's as contrived as any other idea but the information doesn't come out of his mouth, confession-style.
He wouldn't (necessarily) see himself as alternative
Is casual sex the norm (or compulsory) in your fantasy setting?
When you say fantasy I can't help but picture something vaguely Euro-medieval where marriage is about property not passion, and there are euphemisms for not breeding, like going on a crusade for 20 years or joining a nunnery.
There was actually a fine word for it back then: celibacy. It was wrapped up in concepts of purity, religious devotion, and non-passionate chivalry. It meant you were better because you had your urges under control and devoted yourself to higher concepts. (Click the link for cultural information.)
One culture didn't see celibacy as a good, and that was the Romans (and maybe teen-sex movies from the 1980s). Most other cultures on Earth have had an honored and noble concept of the rejection of carnal lust.
After comments from @Galastel and some research, celibacy was potentially a crime against god in Judaism. This is procreative sex within marriage of course. According to the link a 20yo male could be forced to marry – it's really not the same as forcing women to marry and bare children – women died in childbirth, often.
There's less urgency to declare it
An asexual or aromantic person can have sex for purposes of procreation and pleasure, just as many gay people have children using the traditional (heterosex) method. Arranged marriages would be the norm, and not based on sexual attraction. Our modern ideas of sexual orientation are not deterministic. Your character may have children and (several) spouses and still consider themselves celibate.
Where a gay-identified person may experience revulsion at the idea of heterosex (as straight-people may about homosex), it's usually not such a polarity with ace/aro. Instead they typically feel indifferent, unfulfilled, meh it's not worth the effort. Anticlimactic. It's the same way I feel about sportsball games, religion, and the Marvel Cinematic Universe – I see other people are passionate about this stuff but it's like a marketing hoax: they have fallen for hype and are worked up about nothing.
There's not the same urgency for self-defense, to be acknoledged, or the desire to seek others and form a community together. There's also much less urge to rally against romance than, say, someone who has a recently-broken heart, or is nursing an unrequited fixation, or is steeped in anti-sex ideology. Your character will have encountered these and felt indifferent to their rantings too.
A young Hamlet-esque character might sit around contemplating the fear of missing out. But a grown up will accept that the world is littered with drama you can walk away from, and will find a comfortable balance navigating their boundaries. More important, they will have the time and freedom to pursue other interests, so give your character something extra they do enjoy. They may feel out of step with society, but they won't feel as if they've lost anything.
Your fantasy setting may vary. You might need some Margaret Atwood-level social-engineering for societal pressures to impact the character. A meddling matchmaker, parents who demand grandchildren, or an overtly sexual social environment might be necessary before it would even come up.