Rules? No, not beyond any that your publisher or editor might have. But one factor to consider is that, assuming you're not publishing in a specialized or foreign market, your readers probably won't know how to pronounce the words in a different alphabet -- you can't sound things out if you don't know the pronunciation rules. This means that the words you use are just glyphs, not words they can subvocalize and remember.
I saw this used to good effect once (can't remember where) in a story where the characters encountered a sign they couldn't read. It seemed perfectly reasonable that the reader couldn't read it either. But some people might be thrown a bit if they can't "hear" something that's spoken in the story. If you instead transliterate (and change the formatting; italics is common for this), you convey that it's a foreign word while still giving them something they can pronounce.
There are readers who don't care (I know people who can't tell me the names of Russian characters in books, but they know them when they see them), but using the original alphabet will get in the way of people who do care without helping most of your readers. So, assuming you don't have a higher-than-normal proportion of Russian readers in your audience, you will be more accessible by transliterating.
In practice, a few isolated words in a novel won't make a difference either way. But if you have a character who tends to use these words more frequently, you might want to fall back on the transliteration.