Sexism isn't a yes/no kind of thing, and it's a mistake to treat it as such.
Saying that a story or an idea is "sexist" is shorthand. What it means is that it creates, encourages, or reinforces sexist stereotypes, and that those stereotypes have real-world consequences.
So, a good way to come at this issue is to use these:
- What are the stereotypes that my story is relying on, conforming with, or playing into?
- Is my story repeating and reinforcing a particular viewpoint, bias, or narrative?
- Is it a viewpoint, bias, or narrative I support and stand behind? Or am I reinforcing this "by accident" or "because it's convenient"?
- If it's not a viewpoint, bias, or narrative I explicitly support:
- Is it one I oppose? Is it harmful? How harmful is it?
- Is there any change I can make to change things up, and avoid reinforcing it?
- How central are these stereotypes to your story?
Let's try those questions with your particular example: A matriarchal witchcraft-based culture, where the most epic power is giving birth to a magical familiar, at the price of not ever bearing a human child.
1. What are the stereotypes that my story is relying on, conforming with, or playing into?
I can point to several:
- The emphasis on "using up" their fertility plays into real-world views of prizing women for fertility and child-bearing. It could imply that a woman's fertility is her most valuable asset -- and in this setting she has a new way to spend that very valuable coin.
- The requirement never to have conceived before can play into real-world pressure for women to remain chaste and virginal. Even if in your story, protecting a woman's virginity is important for an entirely different reason, the theme of "a woman shouldn't have sex outside of total and utter commitment" is very (pardon the pun...) familiar.
- Treating losing one's fertility as a "price" can be dismissive of both the choices of people who don't want children, and the struggles of people who can't. (How prominent among witches are women who just never wanted kids at all? Is that going to be portrayed as callous or self-serving of them?)
- Magic/power is dependent on biological sex at birth. This implies that a trans or non-binary witch has avenues which are entirely sealed to them, which places a great deal of cosmic, metaphysical significance on their biological sex at birth. "The universe cares what you were born as."
2. Is my story repeating and reinforcing a particular viewpoint, bias, or narrative?
3. Is it a viewpoint, bias, or narrative I support and stand behind?
We've noted multiple stereotypes, so the answer here is to varying degrees.
It also depends very strongly on how, exactly, you portray different facets here. If, say, your portray witches hoping to birth familiars as generally turning to lesbian relationships which don't risk pregnancy, then you've definitely avoided falling into the "isn't it important for women to not have any sex" narrative!
Or, for example, you might worldbuild a society with underlying sexism -- and then note and criticize that sexism. In this case, you're repeating the sexist viewpoint -- but you're not aiming to reinforce it; you're interested in deconstructing it instead.
Even so, you should understand: choosing such a fraught and sex-dependent element, you're necessarily entering those conversations.
And, obviously, the degree to which you agree with any or all of these broad stereotypes is something only you can answer! e.g., Maybe you do think people who don't want kids are very rare and, at some level, selfish. In which case I might disagree with you personally on that issue, but on the level of craft, I will encourage you to follow your own convictions, rather than try and artificially conform to viewpoints you don't share yourself.
4a. How harmful are these stereotypes?
Again -- each one would need to be examined in its own right. And, finding the nuance between what you are willing to portray, and what you aren't, is very valuable in and of itself.
For example, "women shouldn't have sex" is extremely prescriptive and direct, as opposed to the much fuzzier "women's fertility is a really important thing". On the other hand, "women's fertility is the most important thing about them" is pretty awful.
Likewise, "people born with male genitals cannot become pregnant is not a controversial position. "the etheral realm responds only to people born with female genitals" is more iffy. "the etheral realm responds only to people born with female genitals, and actually that's kind of tragic?" is one way to kind of thread the needle -- or at least try to. Here's Neil Gaiman's Sandman back in 1991:
Also note that different people will have different opinions on everything, and different people will have different takes on whether (or how) something is harmful. The 1991 Sandman example I just gave is considered somewhat problematic today (e.g. 1,2)-- whereas in 1991, seeing it in mainstream (and even prestigious) comics was immensely significant. These things are complicated and have many facets.
You're not going to please everyone -- but at the same time, don't write hurtfully out of mere ignorance.
4b. Is there any change I can make to change things up, and avoid reinforcing it?
There always is.
Maybe the etheral realm is fine with trans women, and they can totally do the ritual! They just as much a conduit between the mortal and ethereal plane as cis witches, and they become magic-pregnant.
Maybe the brightest of witches are working hard to discover new ways to summon familiars -- and then it'll be their smarts that are most valuable, not their uterii.
If getting a familiar and then serving society with your newfound powers is seen as altruistic, not power-hungry, then that's no condemnation of people who don't want kids.
And so on. Each one of these is a change to your story, and you might not want to adjust what you had in mind, merely to avoid reinforcing stereotypes that are all around us anyway. That's your own judgment call. Just recognize that even not making a choice, is still a choice -- conforming to existing social norms is also shaping your story, as surely as avoiding them would be.
5. How central are these stereotypes to your story?
The more central an element is, the more care it needs. (On the other hand, the less central it is, the easier it is to just make whatever adjustments make it "safe" and then forget about it.)
In your case, this is the access ticket to epic power, and it's something everybody knows, that informs all of society and its social structure. So... pretty important to get this right, I'd say :P
(This doesn't mean your social structure needs to be a good, kind one. Maybe it's a really sexist society! But then, you probably want to make your criticism of it evident to the reader...)
That's how you do the analysis. You look at what your story is saying; how it's saying it; what the consequences and the implications are. You also look at how much it would "cost" you to change it -- maybe you say, "OK, there's some sexist undertones here, but I can't change it without telling a completely different story, and I think this story is a worthwhile one." That's OK! Not every adventure novel is a paragon of gender equality! Some subgenres have at least a little sexism or bias baked right in! Sometimes you're exploring gender issues specifically, and can't avoid stepping on some people's toes! Sometimes, "forgiving" some sexism is a choice you make -- but it helps if you make it deliberately, consciously, owning up to it, rather than doing it just because you didn't know any better. And, being aware of the issue in the first place will probably make your writing better at this than it would have been otherwise.
These are all your calls to make -- once you know how to make them.
Which brings me to one last and crucial point: In order to recognize sterotypes and biases that you suffer from (along with a whole lot of the rest of society), you need to get outside input. You're not going to know how you might reinforce sexism without learning about sexism. Ditto for racism, gender issues, religious identities, ethnicities and any subculture that aren't your own. Learn about these things in general; learn about the topics you're touching directly in particular, and for elements featuring heavily in your book -- get appropriate readers who can point out issues that you'd never have guessed.
Hope this helps -- all the best!