Don't take the Hero's Journey literally. In my opinion, one of the most important lessons about the Hero's Journey is this: Don't take it literally. The Hero's Journey is not about classical Greek Heros, it is not about myths, it is not about religion, and it is not about fairytales. The reason the Hero's Journey is such a successful structure is that it can serve as a universal descriptopn of change. Each story is about change. Hence, each story can be interpreted in terms of the Hero's Journey.
What is the Ordinary World? In stories such as Harry Potter or Neil Gaiman's Stardust, the Ordinary World indeed is a world in it's own right that is left behind at some point of the story. It is very easy to write stories that are set within the same world and never leave it. So, if you take the Ordinary World literally: Yes, ignore it as much as you like.
The Ordinary World and Mathematics. However, when thinking about the Hero's Journey as a fundamental description of change, the Ordinary World becomes much more general: It turns into the Status Quo at the beginning of the story that has been changed in some manner at the end of the story.
Is it possible to omit the Status Quo of a story? Yes, it is. Lisa Genova did so in her recent novel Inside the O'Briens.
Is it advisable? Hell, no. Each change requires a status quo to be able to judge its significance. Think of it as a mathematical problem: Change in mathematics is described by the differential operator d/dt (or x or whatever you like). When solving a differential problem, the differential operator is inverted, and a differential problem can become an integrational problem. Integration, however, requires at least one initial or boundary condition. If this boundary condition is not specified, it is still possible to find an analytical solution of the problem at hand, but a definite answer to a specific question can not be formulated. Hence, boundary conditions anchor a class of problems to a specific situation and allow a specific answer.
In storytelling, these boundary conditions can be regared as the Ordinary World. They specify your settings and will determine the absolute value of the change that has been achieved at the end of the story. If you do not provide the Ordinary World, your readers might be able to agree that a change occured - but they will not be able to judge the magnitude of this change or whether it was important or not.
An example. The ultimate change of the story is this:
Derek didn't drink for 28 days.
Is that important? It could be. If Derek used to be an alcoholic and the story ends 28 days after he threw away all the alcohol that was left in his flat, this ending is a reason to cheer. If, on the other hand, Derek only used to drink a beer with friends once or twice a week, then no: Going without drinking for 28 days is not especially exciting for Derek. It's still a change, because the longest Derek has gone without drinking before has been two weeks. But is this change worth telling an entire story about?
tl;dr: Don't take the Hero's Journey literally. If you do, it is nothing but a template that will fix your stories into unnatural structures. Think of it more than a universal description of change. If you accept this notion, the answer to you question is: Yes, a story can survive without an Ordinary World. But it would not be the best of stories, since you don't motivate it at all and deny your readers the possibility to attach significance to it.