The number of characters in a novel is probably not a number you can fix. The number of characters in a scene, and in an arc, however, can be significant.
Essentially, each character in a scene should create tension in the scene or arc, should shape the way the scene or arc unfolds. The basic test here is, how would this scene be different is this character were not present? There is obviously no fixed number for this, but there is also obviously a point where adding one more character is not going to add tension to the scene. Even if that new character is in tension, their presence is almost certainly going to take other characters out of tension. Any character that is not adding tension to the scene is adding distraction and should be removed.
The other thing to remember here is that character does not equal person. From the beginning of drama, we have had the concept of the chorus. The chorus may be many person strong, but they speak with one voice. If they add tension to the scene, they do so as a unit. For dramatic purposes, therefore, the chorus is a single character. This does not mean that they could be replaced by a single person; it means that the mob, the people, society, acting in unison act differently from the way individuals act, and thus the chorus becomes a different kind of character, one that acts in a different way and thus creates tension in a different way. The dwarves in the Hobbit are a good example of the chorus.
So the question you should be asking is not what is the right number, but rather, is each of my characters in tension in the majority of scenes that they appear in. If not, they are superfluous and are weighing the story down. Keeping a large number of characters in tension is obviously much more difficult than keeping on a few in tension, but this suggests that the upper limit is determined more by your skill as a writer than by any limit on the reader's tolerance.