There are several different factors here. I will first list them and then go on to talk about them and answer the question more specifically.
- Narrative Problem
- Narrative Tension
- Narrative Ending
These are three interconnected but different things, so it's worth taking a moment to take a closer look.
A Narrative Problem is what the reader is indirectly called to ponder on. This is always* carried forward by the protagonist. In other words, the reader vicariously experiences and ponders on the Narrative Problem through the protagonist.
*The narrative problem is always carried forward by the protagonist, but the protagonist is not always a specific entity or single character (and neither is the antagonist). This is where most people become confused. A narrative protagonist (as a role) can be someone's dark side, a group of characters, or an abstract concept (like, say, an emotion). Furthermore, it can often be so that, where one reader sees character A as the protagonist and character B as the antagonist, another reader will have the exact opposite perspective. Ask people, who is the protagonist in Bram Stoker's Dracula, and see their answers. Finally, a Narrative Problem is not entirely clarified until the very end of the narrative - to this I will come back later, when I describe your own specific case.
When we talk about Narrative Tension, we describe the conflict between protagonist and antagonist - in the way I defined them just above. There can be no narrative without tension; period. A story such as "John met Mary, they got married and lived happily ever after" is about as useful as a raincoat in the Sahara desert. On the other hand, a story such as "John met Mary, they got married and were happy; until John met Anna" is an entirely different thing, because it presents narrative tension.
A Narrative Ending must be certainly inevitable and preferably surprising. Here, inevitable means that it must be a logical conclusion of the narrative preceding it. Here's a counter-example: Think of a detective story that has grown to be so complex, that the author has lost the plot (literally) and introduces a new character in the last chapter, who conveniently brings everything together. This is a Deus-ex-machina ending, and it's an avoidable ending. A surprising ending would be one the reader didn't expect - pretty hard to combine inevitability with surprise, but that's a sign of skill.
Now, let's go to the specifics of what you describe...
First of all, it's entirely plausible (and if you ask me, recommended in certain genres like literary fiction) for a narrative to have an ambiguous ending. Meaning is created by the reader as well as the writer, and the greater the gaps (the less the over-explaining, in other words), the better the reader's chances to supply this meaning.
Can we have an ending where both protagonist and antagonist "lose"? The answer must be "no". But again, most people confuse what this entails. The reason they do is because they become too preoccupied with approaching the narrative roles of protagonist and antagonist in a simplistic, linear manner.
Remember what I said earlier, about narrative problems that don't become quite clear until the very end? This is often a related literary device. Let's see this with an example, it will be easier to understand. In Indiana Jones and the Last Crusade, the narrative problem appears to be who will get the Holy Grail. We have a pretty straightforward case of Indy being the protagonist and Donovan (with the Nazis) being the antagonist. There is tension, related to the narrative problem (getting the Holy Grail).
Nobody does, in the end.
Both protagonist and antagonist fail. At this point, it becomes apparent that the true narrative problem is not the protagonist getting the Grail, but the antagonist not getting it.
The narrative ending is surprising-and-inevitable. It is surprising because you couldn't expect both sides to fail, and it is inevitable because there is no possibility (in the world of the movie) for anyone to gain access to immortality.
So, ultimately, although on the surface the protagonist, Indy, and the antagonist(s) have "lost", the shadow protagonist ("Good"? "Justice"? Take your pick) has won and the shadow Narrative Problem ("How can we stop Nazis from becoming all-powerful) has been resolved.
Sources: 1, 2