I desire to know how to transition smoothly from one paragraph to another in the body text, but there are a number of ways to do it and I don't know which is best. For instance, I could place the bridge sentence in the previous or next paragraph, or I could switch it up. There's also the question of whether the bridge sentence should follow, replace, or merge with the concluding sentence it if it is placed in the previous paragraph, or whether it should precede, replace, or merge with the topic sentence if it is placed in the next paragraph.
If you always transition from one paragraph to the next in the same way, your writing will feel formulaic and boring. If you're writing a 5 paragraph essay and your teacher has given you clear instructions, by all means follow them, but for longer academic pieces (which can be dozens or occasionally hundreds of pages long) always transitioning in the same way may not work.
The key to a good transition is that each paragraph follows logically from the preceding one, just as each sentence follows logically from the preceding sentence. If the order of your paragraphs is well-planned and you have enough time to address all points, you may not need bridge sentences at all. When the topic for a paragraph is not automatically obvious from the preceding one, where to place a bridge sentence depends on why you chose this paragraph order. Is the new paragraph a counter argument to the preceding one? Then the opening sentence should make clear the opposition to the previous paragraph and this will serve as transition and likely also topic sentence. Is the new paragraph an answer to a question posed by the preceding one? Then the preceding paragraph needs to make clear that there is an open question or the new paragraph must point out the hole in the logic of the preceding one. If the new paragraph carries the weight of the transition, then the first sentence pointing out the problem may serve as bridge and a separate topic sentence may show the solution, or you may spend the entire paragraph talking about the hole in logic and the paragraph after addressing that hole.
Transitions require co-ordination from both sides. Once you have your transition written, read it. Read the preceding paragraph and the new paragraph. If the second paragraph does not seem to flow logically from the first, figure out where the flow breaks down and fix that. The fix may require adding a new sentence to close a gap, modifying an existing sentence to make the direction you are going clearer, or even reordering your paragraphs entirely.
Ultimately, Jason Bassford is mostly right. Transitioning between paragraphs is not significantly different than transitioning between sentences, the unit if writing is simply longer.
I think a lot of what some schools teach about how to structure paragraphs in short essays is more to make it easy to mark than because it serves you well in longer pieces. I recommend you bullet-point what you plan to say, going to sub-bullets until you're down to the lowest level. Make those paragraphs; higher-level points can be sections, chapters etc. When I read academic non-fiction, be it aimed at experts or laypeople, it feels like it got all the transitioning it needed just by doing that kind of thing.
I am not sure there is a "most effective" way to transition from one paragraph to the other. Like you said there are many different methods of performing the task. Which one you employ will often vary depending on the style of writing. When writing essays, I tend to prefer placing the bridge sentence in the preceding paragraph. This can improve the flow of ideas when the subject is singular. However, when I write fiction my paragraphs are usually arranged as separate independent units. This free form lends itself better to creative thought.
I see that you have marked this question as "academic writing" but I'm going to talk a bit (ok, a lot!) about other types of writing as well to give you an idea about how you can think about paragraph breaks, in academic writing and otherwise.
In fiction you have at least one hard rule for paragraph breaking:
In dialog, you always write one character's dialog per paragraph.
This is so the reader will always know who speaks, but it can also be thought to mimic how a camera would treat dialog in a visual medium like film or TV.
You want the camera on the person speaking, and when you cut to another person, you also do a paragraph break.
You can use the camera-technique for other parts of fictional writing, e.g. a new paragraph when a description of one object in the room changes to a description of another.
I suggest you think of your academic topic as a room with parts of the topic as objects in that room and your narrative voice as a camera moving through that room. When the camera cuts to another object (or another aspect of that object), you add a paragraph break.
When I blog, or generally write for the Internet I keep my paragraphs (and my sentences) as short as possible.
An internet audience tends to lose interest otherwise.
(Meaning this text generally has the shortest paragraphs and sentences of all my writing.)
I don't think there's a requirement that academic writing has to be boring. And I don't think making the text easy to read would dumb it down. After all, quantum mechanics will be quantum mechanics regardless of the readability of the text.
A single paragraph going page up and page down gives your readers the impression an almost impossible task lies ahead; plowing through your text.
Long paragraphs are depressing!
A reader should be able to look at a page and get the impression that they have several "mouthfuls" of information to "chew down" and that they have been prepared by the author to be just big enough.
Several paragraph breaks on a page also give the impression the writer has the know how to split the text into sections, a bit like a butcher knows how to cut a piece of meat.
You can also use paragraph breaks to make sentences you really want to stress as central to your topic stand out (like the "long paragraphs are depressing"-paragraph above).
I tend to think about my paragraph breaks as a point where the reader can take a breath, so I try to make my paragraph breaks in sync with breathing.
This also means that a bunch of short paragraphs might make your reader start hyperventilating.
I had that experience once and had to put the book down. I've also had the experience of paragraphs so long I "suffocated" and fell asleep... so paragraph breaking is important in this aspect.
However, breathing is also affected by sentence-length, so if you feel you need a long paragraph, keeping an eye on sentence-length, and varying it, might keep your reader conscious...
How do I know when/how to break a paragraph?
The discussion above talks almost solely about the size of paragraphs, however, I hope you've already been able to guess how I separate my paragraphs:
I think the best way to describe it is that several paragraphs can address one topic, but one paragraph should not address several topics. I.e. it's okay to cover a topic using several paragraphs, but will likely be confusing if you change topic mid-paragraph.
As for "bridge sentences": I have never heard of them, and if I use them, it's pure intuition.
Which brings me to my last, and perhaps most important, point:
Reading is writing
You should (absolutely) read texts just like the one you want to write.
This being an academic writing question, I am going to assume (and hope) you have a pile of other people's work as references and sources (and that you've read them at least once).
How do they split their texts into paragraphs?
Can you improve on it?
Do it! ;)