(from comments) my writing is incredibly smart, both in terms of eloquence and imagination
How many female lead characters do you have? Do your female characters have agency and pursue their own life goals? Are the economics of your female characters given the same level of "realism" as your magic system?
Do your female characters have female friends?
Can your stories pass the Bechdel Test, or better: the Delany Test?
Most book readers are women
Hard statistics on book sales by age and gender aren't as easy to find as I expected. Most "data" is self-reported, and the research tends to be skewed depending on who re-prints it. Writers are focused on genre and book classification (romance, non-fiction, etc), while publishers are focused on format (print vs e-books, etc).
Most of what I could find was anecdotal, or journalists reported on research without re-printing the actual data (which is proprietary and costs money). Nevertheless, it's universally reported that women read and buy more books than men do.
Why Women Read More Than Men
Who buys books? 40-year-old women...
Where I found "respectable research" that I could see the actual data set, further digging revealed the test sample was very small. Pew Research conducts a study every few years (skewed for books vs e-readers) but a 2016 survey was only ~1500 people. Compare that to over 1.2 million US comicbook readers responding on Facebook (spoiler: 3/4 comicbook readers are men). Again, it's difficult to check the conclusions when journalists don't share the original research data.
If women don't enjoy your writing, kiss 60% of the book-buying market good-bye
Draw your own conclusions. Women buy and read more books because other mainstream media tends to ignore them. Or maybe men prefer visual entertainment and adrenaline-boosters over reading and thinking. Maybe white men who have the most spending power just don't like identifying with other people and it's holding them back in genres that are about new frontiers and fresh ideas.
YA fiction geared towards teenage girls is very trendy and popular,
and many writers do very well even in spite of rather low quality
writing. You are tempted to, at least for the time being, venture into
that genre and use your skill to start writing to pander to teenage
girls despite really having no interest in YA or romance.
You sound very dismissive of "teenage girls". To include something that interested them you would be forced to "pander" and stifle your "eloquence and imagination". I'm going to give you the benefit of interpreting your question as facetious, and you don't actually believe you can "bad-write" your way to success.
Arguably there is a cause/effect fallacy here. There are no publishing houses run by teenage girls. Publishers are pursuing a demographic that is largely ignored by mainstream movies, mainstream comics, mainstream sports, and mainstream gaming – all of which are a smorgasbord of crappy fantasies for the lowest common denominator male ego. It's silly to pretend that idiot-savants are monkey-typing "popular YA books" without major advertising campaigns and corporate influencers signing the checks. If you do any pandering it would be to publishers, and they seem to pander to the majority demographic that actually pays for books.
You could buck the trend and be hailed as a genius
There are outliers of course. Harry Potter is a YA fantasy series that features a boy, and it actually got more boys to read even though the hero is hardly a manly archetype and his best friend is an over-powered "teenage girl" SJW who can do no wrong.
Black Panther and Wonder Woman made bank repackaging the same old clichés to a demographic that is usually ignored in actionhero movies. These are "exceptions that prove the rule": they play the same tired formula but feature a long-missing protagonist. After both made so much money it's bizarre they have no sequels (yet we have AntMan 2) – it's almost like market forces and audience enthusiasm don't actually decide what stories get funded, instead some corporate bros throw money at projects about dudes they relate to.
It could be argued these outliers are so successful because they speak to a demographic frustrated by being ignored for so long – kind of like the YA "teen girl" conspiracy.
The Reader as "Mary Sue"
Please understand that I relate to your question. We all navigate between cliché and originality, personal and public, ego and id. All of life's decisions are about what to preserve and what to change.
To be honest, teen girl shooting patriarchy in the face with a crossbow sounds just as dumb as magic superpowered ninja druids with laser swords. I understand why people enjoy these dorky fantasies, they "Mary Sue" themselves as the hero so they become the most special/important person saving the galaxy. I agree, it's like pandering to mental children.
I'm not writing mindless Reader-as-Mary-Sue fantasies (that actually sounds profitable, so maybe I should rethink my life goals) but, admittedly, I re-arranged action scenes and added a sapho-erotic angle that wasn't there. I worry that not enough scenes take place in outerspace. I added character flaws just to be polarizing and controversial – among countless other "commercial" decisions. Is it still pandering if you don't flatter the reader with their dumb version of a Mary Sue? And yes, I had to edit out (most of) my author-as-Mary-Sue moments. I'd rather pander to fridge logic post-analysis than serve candy-coated bonbons to morons. Like you, I have considered selling out. If only success was guaranteed, this would be a no-brainer.
If you don't like the demographic that actually buys books, you could try appealing to another outlier group that is largely ignored and hope to hit an unlikely jackpot. These pendulums tend to swing hard after being unnaturally forced in one direction too long. If you write an alternative to the squint-y male power-fantasy medieval Europe with magic and dragons cliché you might win 3 Hugo Awards in a row. Then again, if you make it extremely bad with gratuitous rape scenes you might sell it to HBO – an even bigger jackpot! Ka-CHING!
Or that might be pandering.
Write for yourself, and you'll only pander to an audience of one.