One of the best examples I can think of--where the plot requires a counter-factual setting--is the beginning of Never Let Me Go. It simply states the assumptions the reader needs to accept:
"The breakthrough in medical science came in 1952. Doctors could now
cure the previously incurable. By 1967, life expectancy passed 100
years."
By stating the "facts" that the reader needs to accept, up front, it lets the reader know that an "alternate reality" as is the setting for your story.
If the idea of an "alternate reality" is too strong, I agree with the other answers that you don't necessarily need to draw attention to "the impossible." You just write as if it were possible and move on. There is precedent to this. For example, there is the writing style of "magic realism." According to its Wikipedia page, it has been defined as "what happens when a highly detailed, realistic setting is invaded by something too strange to believe."
Another example I can think of is Dream Boy, by Jim Grimsley. In the book, a youth is beaten to death. From the description of the aftermath, the book gives the strong impression that the boy is dead. Yet he wakes up and walks away from the scene. Grimsley leaves the story ambiguous whether the boy didn't really die, is a ghost, someone else's fantasy, or something else. According to this NPR reviewer, "Grimsley realizes literature is not bound to the laws of the physical world, and he makes the most of this."
In the theatre community, we often talk of the "suspension of disbelief button." Audience members—or readers—will often be willing to push this button, in the service of engaging with the story, if the story is well-told and otherwise believable. From a technique point of view, I think it helps if the situation around the "factually incorrect" event is left a little ambiguous, and that part of the story is relatively fast paced, so the reader isn't forced to dwell on the inconsistency.
In short, I think the answer to your question is in the question itself:
"I can either continue with my impossible (for a non-fantasy, non-scifi setting) idea, or scrap it completely."
Right. I'm submitting this answer to say that "continuing with the impossible" is a legitimate thing. It has an established precedent. The flip side of the coin is that some readers will be less willing than others to plays fast-and-loose with reality, even if it in the service of the story.