Yes, it breaks the rules you learn about writing, but it is possible.
You seem to have a clear reason as to why you want to do it, which is a good start. In the novel "The Queen of Attolia" by Megan Whalen Turner, there's a fantastic example of switching viewpoints in the middle of a scene. She does it (IMHO) so smoothly that the reader barely notices.
It's a bit of a long passage, but bear with me:
He turned. The queen stood at the end of the passageway, flanked by two more soldiers and a third man.
“What do you mean, I'm not allowed on the roof?” said Eugenides, outraged.
The queen walked toward him. The third man, Eugenides saw, was one of Galen's assistants. He glanced from the assistant back to his queen.
“You have someone watching my door,” he accused her.
She looked uncomfortable. Eugenides turned to the guard beside him and cursed. He turned back to the queen, still cursing. The soldiers on either side of her looked shocked.
“You think I'm going to throw myself off the roof?” he asked.
She did. The people in his family tended to die in falls. His mother, even his grandfather. When the palsy in his hands had grown so severe that he could no longer feed himself, he'd been unable to climb to the roof, and he'd tumbled over the railing at the top of one of the back staircases. It hadn't been a hard fall, but enough to kill an old man.
“You started a war without mentioning it,” Eugenides snarled. “You have my rooms watched, and I'm not allowed on the roof. What do I find out next?” He pushed past her and the soldiers. He walked backward away from her. “Tell me you've enrolled me as an apprentice bookkeeper. You bought a lovely house for me in the suburbs. You have a marriage arranged with a nice girl who doesn't mind cripples!” he shouted. He had reached the corner and disappeared from sight still shouting. He was making enough noise to wake every sleeper in that wing of the palace, and he didn't care. “I can't wait to hear!” He spaced his last words out and finally was finished. There was no sound, not even that of his receding footsteps.
The queen sighed and dismissed the soldiers who'd accompanied her.
“Shall I go back to watching his door, Your Majesty?” Galen's assistant asked.
“Yes,” she answered heavily. “Watch him as carefully as you can.”
Returning to her room, she sighed again. The accusation about the arranged marriage had been a home shot. It was a good thing Eugenides hadn't realized it yet.
Notice how it starts off from Eugenides' POV, then shifts to the Queen's. There's a bit in the middle (from the "She did." onwards) that could be from the POV of either of them, and she uses that as the transition point.
I will mention, however, that she only does it once in the scene. I can't say for sure whether or not it would work with multiple changes, but it does make it much harder to continually do it without confusing the reader.
In the end, the final test is always: Does it confuse the reader?
You mention one of your readers being confused - that's a warning. But it doesn't necessarily mean you have to change it. What do your other readers think? Do they notice the switching as well? If only one out of, say, five to ten readers gets confused, then you've managed to pull it off.