I have seen at least a few cases of Latin being used in both fantasy and scifi, and I wonder how it is perceived and how much is tolerated. Being primarily a gamer, the first examples that come to mind are Kingdoms of Amalur and Warhammer 40,000.
Is it a bad idea to use Latin (or similar languages)? What would it be acceptable to use Latin for? names, phrases/expressions, objects?
If I was reading a fantasy or sci fi story in which Latin was the predominant language, I would expect some explanation as to why. Perhaps the Roman Empire had not imploded and instead had gone on to develop space travel and colonized the planets. Or maybe your fantasy is about a kingdom where magic exists and it is in the path of Roman conquest. In either case, as a reader I would expect the author to provide me with a sufficient explanation to allow me to suspend my disbelief. It wouldn't have to be much, but it would have to make sense.
A further complication you would have is that Latin, while arguably a dead language, is still enough alive that some readers would have studied it in school or at least be familiar enough with it that you would have to ensure you used the right vocabulary, verb tenses, etc. Not doing so would definitely affect your credibility as an author.
Using your own created language automatically makes you the linguistic expert and you wouldn't have to contend with the above issues. It is certainly more work, though. Another option is to use an artificial language, such as Esperanto. Harry Harrison took this route in a number of his novels, such as those in his the Stainless Steel Rat and Deathworld trilogy. You might consider reading a few of them to see how he handles using the language. Harrison talked about why he used Esperanto back at the 45th World Science Fiction Convention. Philip José Farmer also used Esperanto in his Riverworld series.
I think there is an issue if you expect people to understand Latin to read it - relatively few do these days, but some will. This means that not only do you have to provide explanations/translations, but these have to be accurate!
I have read some stories where Latin phrases/mottos etc are used, with an expectation that the read will know them - sometimes this is fine if they are well known, but it does provide a barrier to reading. Umberto Eco, IIRC, tends to do this, but then he draws from such a wide range of back-tales, all of which you really need to know, this is just part of my problem.
Using Latin for the names of things or people is not a barrier really - you can recognise the names whatever the actual language is, whether Latin, Vulcan or an Elven language. Using it as the lingua franca of a book raises a whole lot of other issues, not least about language development.
In most cases it's to make the book look more complex, while also being intelligible and cool.
In my own book, you will find loose, slightly-changed Latin sprinkled all over it - like a species of bird called tabbelarees, or "bells." The original word translates to "carrier" as from the term, message carriers (Tabbelarrius Nintius), effectively letting the reader (if they know Latin) know what the bird does for the community. And if the reader doesn't know that, then it sounds cool, it's original, and the bird gets described anyways.
In all, I'd say it's a good use of a deceased language.
I think it works well and can add to a story, if it's used sparingly. Using it for the names of relics, weapons, vessels and titles are all great uses of it that can add another layer of depth and atmosphere to a story.
Since you've cited Warhammer 40k, there are, in my opinions, some Black Library authors who do it well, and other Black Library abominations that don't. The likes of Graham McNeil, Chris Wraight etc use the pseudo latin of 40k quite well. It doesn't distract the reader, it adds depth to the story and can be seen as a little easter egg for those who do understand it to cotton on to the meaning.
And then, at the other end of the spectrum, lies Gav Thorpe. I just recently read The Purging of Kalidus - about the Dark Angels on Piscana V - written by Mr Thorpe. Throughout the dialogue are scatterings of pseudo-latin. Entire phrases that don't have any context, that jar the reader out of the story and add absolutely nothing to it. It's taken it too far and comes across as the author being rather pretentious and allowing him to ignore any failings in the book as a petulant whine of "but you just don't understand"
