To what extent is imagination allowed in literary fiction? By this question, I'd like to know if it is possible to imagine a disease that doesn't exist in reality or that would ruin the work and seem illogical?
I don't believe there is a limiting factor on how imaginative is too imaginative for a story.
My observation is that the limiting factor for story elements that challenge believability is understandability, which I've seen addressed in two methods.
The first is the author presents the imaginative element in a way that the story builds up its representation so that the imaginative element makes sense in the story. Harry Potter is a good example of that. While we don't get a sense of what magic is, it is shown operating in the story and it seems to have largely predictable effects and limitations -- thinking about the books here and not the movies. Magic needs a phrase and a wand and the proper mental attitude and viola, Levitating Feathers!
The second that I've observed is the author just drops it on the table, in a take or leave it kind of way. Characters in the story take on the role of arbiters: some declare the thing misunderstood and others declare it a miracle. The book I'm reading now, Ursula Le Guin's 'Lathe of Heaven', is an example of this. George Orr has the ability of effective dreaming -- he changes reality by dreaming. The antagonist, Dr. Haber is a psychiatrist who doubts George's claims, believing instead George is suffering from a delusion. The events of the story -- increasingly outlandish -- convince Haber (and by extension the reader) that George is in fact telling the truth.
Why should there be a limitation? In my opinion, the more imagination and creativity that you possess, the better your work will be.
Depending on the genre of your story, 'imagination' as you define it will vary. With fantasy stories and sci-fi stories, "The Sky's the Limit." When you have something like a romance or a mystery or something like that, it really depends on the setting. I mean, you could have either of those genres occurring in our world, in a real location at a real time. Or you could put it in a land like Oz or Middle Earth.
If you are considering historical fiction, there will be your strictest limit. You can't very well add an army of dinosaurs into something like the civil war IF you are writing pure historical fiction. But I have read some books that literally had dinosaurs in the civil war, and it was perfectly fine.
Never think there is a limit for imagination. The limit is only what you set it at, and be hesitant to think that someone can tell you otherwise.
Fiction is supposed to get the reader to imagine something that the writer has imagined. Writers may have much more capable imaginations than an average reader, but if they are good at describing their imagination in words then they can lead the reader to imagine it, too. So a sufficiently skilled writer is not really limited by the reader's capability of imagination.
On the other hand, there are some things that readers don't want to imagine and will therefore choose not to. For example, I don't really want to entertain the idea that slavery could be morally acceptable, even if it's "only" fiction, and I think many other readers don't want to imagine that either.
Put another way, "getting the reader to imagine something that the writer has imagined" is a cooperative process, so the reader's willingness to cooperate is a kind of limit, particularly when there are things that no amount of writing skill could convince the reader to cooperate with.