Personally the dissonance whenever I have imagined a character for hours and maybe thought about their stories throughout some days because I can't read a book straight in one go is the biggest problem. It's very irritating because some part of me wants to scratch all that I have thought about through the time and rebuild it to have the same image as the author, or at least something roughly close, and another part just wants to continue with the image I had in my mind because I am used to it and probably have come to like it with regards to the story.
This is not just happening with their general appearance, but also with other characteristics. As someone who is right-handed it can be irritating to follow a character's story for hours and in a critical moment their "strong hand" is bound or something like that and instead they use their right hand - I thought right was the strong hand. It's a bias, I know, but I will imagine every character to be right-handed by default unless explicitly told that they are left-handed. And if it becomes important which hand is doing what it will be irritating.
If it gets important you should mention it at an early point in time when a "viewer" would have realised it. There is no point in talking about the hands if you are only concentrating on the feet in a scene. But you can just use a calm scene to showcase this by making a generic allusion to the character using their left hand for something like holding their coffee cup.
It's the same with their clothing, general appearance, mannerisms, ... You don't have to scream in the face of your reader that the girl's hair is 76 centimetres long at the longest point and has a mahagoni colour when seen through the light of two of the three suns of your fictional planet around noon, but only if it's still a bit wet from the shower because otherwise it would be brighter - just say that she has long, brown hair. And once someone in your book wants to talk more specific about the colour, like when they are comparing the hair colour of two characters, you can get more specific. But if you never mention that she has really long hair and that will be relevant, for example in a fight scene, your readers might be confused if they imagined someone with short hair.
Giving some rough details when everything is new and filling out the details throughout the book is always a good way to keep your readers attention and not get derailed, but still making sure that they understand what is going on.
People think in the form of pictures. They will picture the scene you describe. Everything you don't describe will get filled in by them so that they can still have a picture. The longer you wait to "fix" things in the picture the more pictures will have accrued that need to be "fixed". The appearance of humans is incredibly important for us humans to determine whether we like or dislike someone, which is probably why your beta readers or other feedback-giving people told you that you need to give your characters a "face". If you don't give them a face and clothes and a general appearance your readers will do it for you to the best of their abilities. But the characters that are acting are always in the middle of the picture. If you haven't described them it will feel like you have "forgotten" something that now needs to be filled explicitly.
As long as nothing important happens in a chapter it would be okay to leave the generic description for the end of the chapter. Most of the time a very rough description will happen when a character first appears, but there are all sorts of reasons why you might delay the description. Wearing a helmet, darkness, other things that are more important, such as death traps, ... Anything that makes your character later say something like "Now that I could have a good look at him/her/ ... I realize that [whatever you want to say to describe them]."
Waiting multiple chapters is too long most of the time. At least for the generic rough description. It's not a problem to wait with the detailed description for a few chapters though.
All of this obviously depends a lot on the story. If the appearance is important the moment the character appears it needs to be told. If it's not possible and relevant to describe you should not waste too much time while other things are obviously more important right now.