Publishers are in the book marketing business. Their job is to figure out which books will sell and how to sell them.
There are three reasons why a publisher may reject a book:
- It is not good enough to sell
- There is not a big enough market to sell it to
- They don't have the knowledge/channel to sell it to its target market
Before the Web, the only way to make money selling books was to make reasonably high sales to a reasonably well defined market in a reasonably short period of time. Otherwise warehousing and distribution costs would exceed any reasonable expectation of profits.
The Web changed that equation by reducing warehousing and distribution costs to near zero. This has given rise to a phenomenon known as the long tail. There have been several studies looking at how the long tail has affected Amazon's business model. The long tail is the vast collection of goods which individually sell only a few copies a year, but collectively add up to a huge amount of business. Apparently the long tail accounts for half of Amazon's business (but don't quote me on that).
The long tail can't exist in the bricks and mortar world because of carrying costs. It tends not to work for traditional publishers and record companies either because of the high fixed costs of publication. But it has revolutionized independent music, allowing thousands of bands to build small followings. Not enough to make them stars, but maybe enough to live on, or at least generate a decent amount of mad money.
Self publishing today, therefore, is not the same as self publishing in the age of paper. It is about getting access to the long tail for niche works -- works that will sell slowly to a specialized audience that a traditional publisher would not know how to reach or could not afford to address due to low volumes.
If you are writing a long tail work, and if you write it well, and if you have a platform to promote it, you can probably make some change with self publishing.
But that is not what the vast bulk of self published work is. The vast bulk of self published work is work that is entirely mainstream in its intent that the writer either could not make good enough to sell to a mainstream publisher, or just happened to be in an oversubscribed genre that is not selling well at the moment. The wide sargasso sea of this stuff is not making anybody but Amazon any money.
Occasionally, of course, some work that the traditional publishing world did not think would sell, breaks out of the long tail and becomes a big success. It is very very rare, but it happens.
There may also be a few in the wide sargasso sea that are eeking out some cash through sheer volume and grit, but unless you can match their energy and gall, that is not a road you can easily go down.
To address your question, therefore, the right approach it to find the right market for your work based on its intended audience. If that market is the market served by traditional publishers, then that is the right route to take and you need to keep working on it until it is good enough to sell (or until some event occurs that increased demand for work in that genre).
If it is addressed to a long tail kind of audience, either find an appropriate niche publisher or go the self publishing route because a traditional publisher is not going to publish it no matter how brilliant it is. But don't go this route as a way to stop short of making the book the best it can be. A book has to be better to sell in the long tail market because it does not have the imprimatur of a major publishing house to recommend it.