1) Traditional publishing with an agent
Pros: You as a writer focus on only one thing: writing. You have an agent who is responsible for shopping around your manuscript. Once it sells, the publisher is responsible for all the overhead: editing, printing, selecting a cover, distribution, marketing, and sales. There is no denying that there is a certain level of recognition that goes along with being traditionally published. It is like having a badge that says "I made it"! This is something that shouldn't be ignored.
Cons: You must first find an agent, which will require sending out query letters and inquiries. Most agents will only represent established writers, so for a new writer, this in itself can be daunting. If you choose to skip an agent, then you still have to go through the process of shopping your manuscript. Your royalties on sales are very small, as low as 6-10% for a new author, perhaps as high as 20% for a truly established writer. The author has little or no input on the cover selection or layout of the book. The time to market is generally 12-18 months. If you receive an advance, you don't collect any royalties or further payment unless and until your book earns out the advance. Your book will usually remain in a book store for 6-8 weeks before it is returned to the publisher, so you have a very small window of opportunity. Regardless of how much money you actually earn in the end, your agent will collect a percentage based on your contract agreement. That could be anywhere from 10-25%, but is generally closer to 15%. Traditional publishers tend to price their e-books at or near the same price as their paperbacks, which makes it difficult to compete with self-published book pricing. Also, although traditional publishers receive a much higher profit on e-books, that rarely equates to a higher royalty percentage for the author.
2) Small publisher
Pros: See above (with the exception of the agent shopping your manuscript)
Cons: See above (minus the agent comments). Also, a small publisher will have less money to spend on promotion or marketing. They will also more than likely have less clout with book retailers, which means that your book will have a smaller distribution and will likely spend less time actually in the stores.
3) Self-publishing (e-books or "Print on Demand" - POD)
Pros: The writer has full control over the selection of cover art and the layout design. The writer does not have to share any royalties with anyone. The royalty rates for e-books is 70% (based on the Amazon KDP program, with books priced between 2.99 and 9.99). The royalty rates for POD trade paperbacks is dependent on whatever price you choose to set for your book. (The author essentially controls their royalty rate.) The time to market is less than 30 days, regardless of whether you choose e-books (which is closer to one week) or POD. If you find errors in your book or decide to change any part of it, you can do so virtually overnight, whereas this is virtually impossible with a traditional publisher. You have the potential to generate interest in yourself as a writer, and if you are successful in selling at least one of your books, then you have an advantage of other writers if you should decide to go ahead and pursue a traditional publishing deal.
Cons: The writer is responsible for all the legwork to get the book ready and to get it to sell. This means you have to either do the editing yourself or hire it out (which could cost a couple hundred dollars, depending on the size of the book). You have to either create your own cover art or hire that out (generally costs about $100). You have to do the layout and design to create the final format for your book, and it usually requires at least three different formats, or you can hire this out (usually for about $50 or less). The writer has to take responsibility for sales and promotion, which means figuring out a way to draw attention to the book and persuading people to buy it. The time involved with having to promote your own work and do the networking to try to draw attention to your work can be very debilitating to your writing. One of the biggest challenges for most new writers is finding the time to write, but if you have to spend the same or more time promoting your work, it becomes even more difficult to get anything new written. There is still a stigma attached to self-publishing that causes people to frown upon authors who opt for this choice. There is the possibility that your work will be perceived by some as inferior or inadequate, and until and unless you can generate the sales to prove otherwise, it may be difficult to rise above that perception.
4) Vanity publishing
Pros: Similar to a traditional publisher in that they provide the editing (minimally), and they do the design, layout, and printing.
Cons: Could easily cost as much as $3000, depending on the number of books ordered and the size of the book. They do not do any distribution or promotion of your book. The writer has to find a way to get people to buy the book. Most book stores, and also most online retailers, will NOT accept books directly from authors if they go through a vanity press. Royalties tend to be low because you have to price the books higher to recover the sunk costs.
5) Creative new approaches
I can't really give any pros and cons on these because they are, after all, new! However, I know a number of authors who have done serialization of their novels on their web sites. They basically post a new chapter each week, while also providing readers with a link to purchase the entire book in case they don't want to wait.
I also know a couple of successful authors who originally started out by providing their entire novels as a collection of podcasts. Once they reach a certain level of subscribers, they then sent the subscribers a link to purchase the complete novel.
I am currently working with a group of writers to create a new novel in which each writer takes a separate chapter and builds the book from scratch. It's challenging, but so far the book is turning out better than I had imagined.
I also know of a group of writers who went together and created a short story collection where each writer provided their own story as well as sample chapters of other books they have written. I believe there were eight authors involved, and three of them have now been signed to traditional publishing deals!
Disclaimer: While I will be the first to admit that I strongly believe that the potential for success for a new writer is much greater through self-publishing, I also believe that the recognition of having been published by a traditional publishing house will garner a writer much more attention. However, I am seeing more and more examples of writers who strongly believe in their work who have managed to land traditional publishing contracts after they have proven that they can sell their own books. It is my opinion that this may be a faster avenue for obtaining the more lucrative advances and esteem that come with a traditional contract.