You seem to be worried about "defamation" which consists of publishing statements that are 1) false 2) reasonably (and wrongly) believed to be true) and 3) are "highly offensive" to a reasonable third party.
First, truth is a defense against "defamation" (although you can get into trouble on other grounds if you publish private facts like someone's social security number).
Second, labeling something as fiction is a defense, although an imperfect one, against people believing that it is, in fact, true. Basically, you have to avoid being so factual with part of your story that people believe that the rest is also factual.
The third, and perhaps most important issue, is that the target has to be defamed in the eyes of reasonable third parties. So something like Mr. X thinking, "I met with Virginia for two hours and I think she's describing me" doesn't constitute defamation. What would constitute defamation is someone going up to Mr. X and saying, "I'm not going to hire/do business/associate with you because of what I read about you in Virginia's book" (which happened to be untrue). No reasonable person would do this unless your book described Mr. X to a degree of "granularity" that it would make it practically impossible to believe that your character is anyone else.
And even the "defamation" has to pass the reasonable person test. A suit by a former beauty queen was thrown out because the book accused her of having sex while levitating in mid-air, a physical impossibility. In the late 1970s, a judge threw out a suit by someone being called a "bastard" by noting 1) the term was in common use and 2) one of out of every four Americans was then being born out of wedlock, meaning that someone could not reasonably be defamed by a term that accurately described that many people.
I am not a lawyer and am answering as a writer, and prospective juror.