As a frequent beta-reader, often for friends, I struggled with this question -- and I'm pretty happy with the answers I've figured out.
Don't pronounce judgment; help the author level up
It makes a lot of sense to tell an author, "Listen, I don't think this is ready for publication." Because, really, isn't that the most important thing?
Well, it is, kind of. But the problem is, it's not very helpful. As a beta-reader, your goal isn't for an author to publish or to give up blindly, on your say-so -- it's to help the author understand better themselves. You've done a good job if the author understands your issues with their book -- not just a thumbs-up/thumbs-down final grade.
And here's the magic of that: If you've done a good job explaining what the issues are, you don't need to say "this isn't ready yet." The author will understand that on their own. They'll see problems that didn't register before; they'll want to fix them.
That can still feel overwhelming and harsh, to be sure. Which is why I find that:
Detail is better than sweeping statements
A comment like "I didn't connect to the protagonist" could point to a serious problem with a book. But as a criticism, it's kind of broad. Particularly with an inexperienced writer, they might simply not understand what that means, where it's coming from, or how to fix it.
What I've found is that the biggest problems in stories are often hard to pin down -- they're not any one thing or a particular section; they're the accumulation of tiny issues into something much larger. "I can't connect to this character" might be the accumulation of moments like "Wait, doesn't he have a day job?"; "This bit seems really inconsistent with how he was behaving ten minutes ago"; "How come he never mentions having a family or friends?". Each is minor on its own; but together, you can see where the sense of "not connecting" is coming from.
So: Try and explain, to whatever extent possible, where each point of criticism is coming from. Key your criticism to the text. This makes your points easier to understand and digest; and it also makes them more fixable. This might feel like dumping more criticism on the poor author -- but if you do it well, you're breaking big harsh writing issues into smaller, bite-sized morsels, that the author can grasp and consider and fix.
I've written in the past about how I do my own beta reads. This might not help you this particular time, but I feel like this has been such a help to me: My beta reads always include notes I jot down as I read. Notes just capturing my immediate reactions. Not "fix this," just "I'm feeling this right here." Why is this helpful? Because it documents my own feelings and reactions, in a very detailed way. And feelings are pretty persuasive -- you can argue with analysis, but you can't argue with the fact that I felt X when I read Y. And this gives me two things: (1) the author gets insight into my reading experience, and (2) I can refer back to my reactions at a specific point when, at the end, I do want to make some larger point. This takes a fair amount of work, but it's very effective and I highly recommend it.
It's very easy to come across as dismissive or condescending, when you're giving detailed criticism of somebody's work. That's a bit beyond the scope of this question, but I highly recommend the Critters Diplomacy page for great, concrete details of how to communicate criticism diplomatically.
Identify the story's strengths -- and its goals
A piece of advice you'll often hear is, pair negative criticism with stuff you did like about the story. This really is crucially important -- as a critical reader, the bad stuff can really stick out at us, but pointing out what the story does do well is just as important.
(And, it's definitely worth acknowledging the accomplishment of writing an entire book or short story, even just a first draft. That really is huge.)
What's no less important is identifying the story's goals. Is it trying to be a zany romp? An affecting romance? A pulse-pounding thriller? Even if you feel it's done so poorly -- maybe especially if it's done so poorly -- point out the goals you've identified. Tell the author, "I see what you're going for here."
- That places you firmly on the author's side. The author may not have done a great job accomplishing their goal, but by goodness, you understand what they're trying to do, and your advice is there to help them accomplish that.
- This gives you a lot of positive things to say even when the story is very weak. Even if nothing in the story is working for you, the author's intentions are good and you can appreciate them. This lets you strike a much more positive tone -- not "this is really bad", but rather "this is aiming at something really good but it isn't there yet."
- Lastly, sometimes a beta-reader can be way off-base with their reading and viewpoint. Maybe they don't click with the author's style; maybe they're beta-reading in a subgenre they really don't get along with, or are unfamiliar with. If you've mis-interpreted the story's goals, the author will understand that you're reading from an entirely different viewpoint. They'll have a much easier time understanding your comments as coming from a reader who was looking for something else -- and that's a really good thing to know.
When you max out -- stop
Sometimes you read something and despair of it long before the finish line. Maybe it does something that turns your stomach and you can't read any more of it charitably; maybe it does one annoying thing again and again, and all your comments are going "STOP DOING THE THING ALREADY".
You are not obligated to beat the story into the dirt, or to chronicle every fault and misdeed it contains. Your goal is to help the author improve, not to tell them how bad they are. By commenting up to page (X), you're giving them helpful, valuable feedback -- the work they need to do to fix it, if your comments are good ones, is already plenty.
And the simple fact is: if you reach the point where you're hate-reading a story -- where you're getting nothing out of it except enumerating its faults -- then you're not in a great frame of mind for giving helpful, constructive criticism. Your points may all be good -- they may be great! But if the underlying story is "I am sick of this story 30% of the way through," then offering tucks and polishes for the last two-thirds might cost a lot more than it's worth. For you, as a frustrated reader; and for the author, who gets a bunch of grumpy notes piled on top of the more fundamental criticisms. Better to kindly and constructively say "I stopped here; here's what I think you can improve", than to wring every last problem out of a story you can't stand.
(I'm not saying "abandon a book that's sub-par"; there are plenty of books that aren't ready, but can still benefit lots from a full beta! The difference between "this book is not ready" and "I hate reading this book; it makes me angry" is important.
Also, some structural issues are really hard to address without knowing where they're headed at the end -- those might require special care, or just flipping ahead to the end.)
The bottom line is: Writing is hard. Much harder than people often realize at the outset. And, yes, that first thorough criticism can be a disheartening experience.
You can't really shield the author from that (and you wouldn't be doing them any favors by trying). But what you can do is approach this as a supporter and a guide; trying to help them see the scope of the task -- and what paths exist to do it well. Be on their side; encourage them to overcome the difficulties even as you are unflinching in pointing out what those difficulties are.
Best of luck to you. You sound like you're making a great effort to help out writers both helpfully, and compassionately. Kudos, and all the best.