I was thinking of writing a story that starts out as a serious drama, but suddenly shifts to a comedic tone without any clear reason or transition. I thought it would subvert expectation and make the whole story more funny, but I am not sure if this is a bad idea, because I've been told many times that you need to be consistent with the tone and style of writing and a sudden shift could be jarring for readers and may take them out of the story. How can you make this work?
While there's nothing to say you can't do this, it's unusual for a reason.
When a reader selects and starts reading a book, they become invested in it (hopefully!). They say that the first 20k or so of a book is almost you teaching the reader how to understand your style and story.
To suddenly flip the table and do something entirely different is likely to lead to a bad reaction. If they're invested in the serious story you're telling (and if they're not, they probably stopped reading already) then you dropping a giant switch of tone on them is unlikely to be well received.
Again, this is not to say you can't do it. Anything can work if it's well executed enough, but you might want to think seriously why you want to do it, and whether anyone else wants to read it. Or perhaps you just want to try for your own interest, which is fine.
One suggestion I'd have if you want to mix in a more lighthearted side to the story is to switch POV character - you could have a more serious, angsty character, and a more comedic foil.
Expanding on Phil's answer:
He's right that you run the risk of alienating readers. But maybe that's okay. Much like genre, tone sets the expectation for your reader and is something of a contract; breaking that contract could be a fun and intriguing surprise, or it could backfire and disenchant your reader. I also agree with Phil about this needing to be well executed. Success here demands good writing, maybe even great writing, and you may still lose readers. But maybe that's a moot point: why would anybody ever write something that isn't good or great?
Ultimately, and this may sound romantic, the answer to your question is really a different question, i.e. what is your goal in telling this story?
Are you looking to predictably sell copies? Then stick to the script for best results. Get your coin and eat well, slainte mhath!
Are you an iconoclast or artiste? Then fie to the rules. Style, genre, word count, cover, blah blah blah... that's all just dogma. Or marketing. Helpful for determining where a book should be placed in the library or for search optimization, but not for telling a story.
For me personally, I prefer the latter. I never ask "what are the rules", rather "what is the story I want to tell"? Tell that. Tell the story you envision, the story of your characters. Telling your audience what they want to hear is a path to mediocrity: vanilla and palatable. Unsurprising = boring. Blech. If you're not writing with passion, vision, excitement, truth and even a bit of swagger, then don't expect your reader to respond with passion or excitement.
Also, if you have a strong vision, and this is your agent or editor telling you something contradictory to that vision, then find a new one. Somebody who will ride; who is a fan of your story; who provides helpful feedback. Not an arbiter.
I think this comes down to something similar to what Brandon Sanderson describes as promises and payoff (YouTube link), except where he is discussing plot, you are talking about tone and style. But I think the ideas of promises and payoffs are also applicable here.
The reader is going to be disappointed if they feel that they were promised something that they didn't then get, and what they got instead is not strictly better than what they were promised. So if the first half of a book promises a serious drama and then the second half doesn't give a suitable payoff for that, the reader will be disappointed.
So how to make the reader not disappointed by the switch? It seems to me there are only two options: either make sure the reader isn't expecting a serious dramatic conclusion to the story, or give them one despite the tone shift. Or the hidden third option (there's always a hidden third option), do both.
Option 1 is about setting expectations - if you promise a jarring tonal shift, then when it happens you are keeping your promise rather than breaking it. For example, if the book is titled Pride and Dumber: A Drama and A Comedy, readers could know going in that there are two separate parts with different tones. You could also give a small taster of the comedy writing in the introduction before the drama begins, so readers are primed to know that there is comedic writing in your book. Another thing worth considering is typography - for example, when a chapter begins with an epigraph, it is often in a different writing style, but the reader expects a different style because the epigraph is formatted differently to the rest of the chapter; the same might be possible for a whole section of your book.
Option 2 is about delivering on the dramatic conclusion regardless of the tonal shift. This could be because the book is written as two largely self-contained stories, so the drama payoff happens before the tonal shift, and then when it happens, the reader hopefully feels like the drama is completed rather than abandoned. Or the payoff could be made in a shift back to drama at the end of the book, or perhaps a second shift to a more high-brow comedic style that would allow elements of drama to fit in. However, if you do this then give the reader some indication that, despite the tonal shift, the promise of a dramatic payoff is still going to be kept.
I don't have much more to comment about secret option 3; this would mean you prime the reader not to expect the serious drama to culminate in a fitting dramatic conclusion, but then at the end you give them that conclusion anyway. This is like Sanderson promising someone a toy car and then giving them a real car - it's OK to not give the reader exactly what they were promised, as long as what you give them is a better version of it.
Flowers for Algernon is a serious yet gentle story where the author indeed changes style through the story. It was a hit, won awards and the clever, tactful use of this change in fact adds to the magic of the story imo.
I thought it would subvert expectation and make the whole story more funny
That is cool. If the reader actually wants that. But consider that a reader looking for a funny story might be bored by the start and won't really reach the funny parts.
And someone expecting a serious story might be annoyed that you change the story into something trivial after them having invested so much.
I think for a short story / joke it's okay to do this sudden shift in tone. But for a longer prose it would be harder to pull off.
You might be interested in the trope "Bait and Switch"
Tone and style should be chosen for effect.
It is possible to change tone. This should correspond to something happening in the story such that it makes sense. A new character entering, or leaving, is a frequent such thing. Or if some event occurs, or during the process of an event.
Consider: A couple are expecting, there is a tone of preparation and the usual things about not letting the mother do various things. Don't run up those stairs mom-to-be. Then the baby is born and the life of the parents changes. The tone of description could well change.
You suggest switching to a comedic tone. If a comedic character enters at the time it might well make sense. If a comedic action begins, likewise. Maybe the neighbors have put on music and it's Dolly Parton playing Yakety Sax (the theme from Benny Hill).
To just change the tone with no apparent reason would be jarring. It is not impossible that there is a reason that simply has not been revealed. To pull an example "out of the air" maybe somebody is piping in laughing gas, and the characters are all getting silly as a result. Or maybe the aliens have turned their Absurdist Humor ray on, but nobody realizes it. You would need to payoff the mystery later for it to work.
In the essay Fenimore Cooper's Literary Offences, Mark Twain lists eighteen of the "rules governing literary art in the domain of romantic fiction" that he believed to have been violated by Cooper.
One of the rules he lists mandates consistency in the style. I won't quote it here on account of the racist manner in which Twain worded it.