I am wondering if there are any rules for misdirection like how would you tell if a misdirection was done right or wrong in a story? Are there do's and don'ts or a point where you can over do it when misleading readers and if so why?
This should follow the same rules as for making a good plot twist.
A story without a twist (or up to the twist point) should be also compelling and interesting to the reader.
Misdirection should not contradict "direction" - they both should look valid from different point of view.
Audience should be more satisfied with post-twist direction rather than original misdirection.
If you look at the classic examples of misdirection, like "Fight Club", or "Sixth Sense", you see that lot of the plot is lined out to develop "misdirection", but then switch to "main direction" feels rewarding rather than upsetting.
Kurt Vonnegut is famous for saying,
Readers should have such complete understanding of what is going on, where and why, that they could finish the story themselves, should cockroaches eat the last few pages.
Misdirections can be helpful because the reader doesn't want it to be terribly obvious what's going to happen next. However, the real direction the plot is heading should be more sensible/compelling than the misdirection, if the reader were to stop and think about what you've told them about the character(s) and plot.
Robert Heinlein said ‘the best way to lie is to tell the truth so no one believes you.‘
I think this applies to foreshadowing in storytelling. If you can foreshadow what is going to happen in your story, but the form and structure ensures that it will be taken out of context and misconstrued, then you have a wonderful misdirection that will hold up at the end of the story.
As far as rules of misdirection,
Only unreliable narrators can lie so by contrast your reliable narrator can never deliberately mislead the reader. They can be wrong and make mistakes, but that means the narrators discovery of their error needs to coincide with the discovery of the truth. I think this is difficult to do with out it feeling like a cheat.
You can as much misdirection in your story as you want as long as it doesn’t interfere with telling the story.
The best misdirects are the small aside events in a story that be confused with revealing character or establishing setting and scene.