Some people write books about serious things and real things in life. If a book is written like that, how can you tell if it is a novel or a documentary [non-fiction]? .
Sometimes it may be difficult to tell if a book is based on real events, but often the book makes this explicit.
"Documentary" is a term generally applied to films, not books. But there are books based on true events, and they're generally just called "non-fiction". There may be references to "novelistic form" or "literary non-fiction" to describe a book in novel form--i.e., with description and dialog, treating the story and people as plot and characters.
"Novel" is generally used to describe a fictional book. Novels may be based on real events to varying degrees. They may even be extremely thinly disguised.
The only way to know the difference is how the book is described: It may be shelved as fiction or non-fiction. There may be clue in the introduction. But how closely a novel is based on real events isn't always known.