The fact that you have never seen an ad for a book should tell you something. Advertising doesn't sell books. If advertizing sold books, you would see lots of advertisements for books, because publishers would do what works. (As the comments indicate, publishers do certain amounts of targeted advertising in some markets. Nothing so far indicates that ads help an unknown self published novel, however.)
What does sell books? I'm no expert, but as far as I can tell, it comes down to word of mouth, hand selling, and bookstore placement.
Word of mouth happens when a few people read you book, love it, and tell their friends, who then tell their friends, etc. etc. You only need a few initial readers to spark a word of mouth expansion, but you also need a really good book. (There are stories of books that languished unnoticed for years before a word of mouth expansion suddenly vaulted them to prominence.)
Hand selling means that someone -- the author or a bookseller -- personally talks up the book to a customer, usually in a book store, but possibly in another venue (I have hand sold my technical books at industry conferences). Hand selling a book yourself is, of course, incredibly expensive and time consuming and is only really going to pay off if the book gains enough momentum to start a word of mouth expansion and to inspire hand selling by book sellers.
Bookstore placement means that your book gets on the rack at the front of the store, gets placed on an end cap, or turned face out on the shelf -- anything to make it more noticeable to a reader who is just browsing with no clear idea in mind of who or what they are looking for. You get bookstore placement by being published by a major publisher with a lot of clout with bookstores who is willing to promote your work through their sales channels. In other words, if you self publish, you don't get bookstore placement (except maybe from your local independent bookstore if you chat up the owner).
There is also reviews in prominent publications, Oprah's book club, and the major literary prizes. But a self-published book is not likely to get within a country mile of any of those. The major publishing houses own the pipelines to those places. You would need to already be a major self-publishing success story to even have a chance of being looked at by any of these.
When it comes to seeding your word of mouth campaign on social media, I don't think the necessarily costs you anything more than time, but my impression is that you are more likely to succeed in pitching your book to your already established social media following than you are to build a following around the book itself, at least until the book itself, or you as an author, already have an established reputation.
In short, if you want to be successful self publishing, I think you have to be prepared to do a lot of hand selling. And, of course, you have to write an exceptionally good book.