I have the exact same problem with the English language. Although, to make it worse (or better), I have an European character (who studied British English) living in the USA. Whenever the chapter is written from the European character's POV, I do not have to worry about the mish-mash - it's part of the character's speech pattern (other characters have pointed out some British words, like the infamous 'a rubber', so it's all good). But I have a lot of chapters from American characters' POV... oh, and the occasional actually 'British' chapter.
Here's the solution:
1) make a cheat sheet
Go online and mash together all the British vs American English words lists you can find (not all of them have the exact same words). Mix it all together. Do the same for the idioms, though that's not as common. Study them. Just picture yourself back at school, yay!
If you attained your English fluency with books and films from Australia and Canada, add those English varieties to your cheat sheet. Moreover, add to your list any regional particularities you come across, even within the US (eg. you all vs you guys... vs even the Irish ye).
2) invest on a spelling checker that differentiates the two
I believe you'll have to pay for one though. On the other hand, you won't have to worry one bit since the app will do the hard work for you.
3) go for 'universal English'
This approach is the trickiest, as you won't find as much information. The idea is to avoid words and idioms that belong to a definitive variety, favouring the ones most used by non-native speakers.
 4) use NGram viewer
Whenever you want to quickly check how American or British a word or expression is, go for the NGram viewer.
You mention you write using British spelling, the American one feeling off. To me, this means you probably studied (and primarily learnt) British English. There are a few grammatical differences between the two - a good university level grammar for language learners will explain those differences - so you may want to check what comes natural to you in that level too.
My advice for your particular case is to write in the variety you feel more comfortable with. Let the publisher worry about 'translating' it to the other variety when the time comes (if need be - British and American are really not so different if you compare them to other languages such as European and Brazilian Portuguese).
Where it comes to the characters' speech patterns, you may wish to make them sound 'mish-mashy' on purpose. Theoretically, it might signal that the fantasy language is not a 'real earth-country specific language'. Of course, that would have to be pointed out in a 'before' or 'after word' and, naturally, a few native speakers would have to tell you how odd it might feel. On the other hand, you may wish to have characters from a certain area sound more British, and others more American.
But do write in the variety you're more comfortable with. If that variety ends up being the mish-mashy one, so be it. In that case, worry about BrE vs AmE only when editing - and the app that picks up English varieties will be a HUGE help. Believe me. I know.
While I can't speak for how odd an English mish-mash will sound to native ears, I can tell you that mixing two varieties of Portuguese is incredibly jarring. Of course European and Brazilian Portuguese have very different grammar structures ('saw me' is 'viu-me' in European and 'me viu' in Brazilian) and a greater number of vocabulary changes (lift vs elevator kind of changes). Still, assume there is a level of 'jarringness' and avoid the mish-mash unless there is an in-world explanation (a character's speech pattern, a fantasy world's speech pattern). And, if you do go for the mish-mash, then avoid aggressively American or British idioms (that is, ones which aren't often used in a world-wide setting and that are opaque for most people outside the country originating the idiom).