I am not a lawyer, so this is not legal advice, but my understanding as an author is if somebody else can convince a jury that, for people that know the real person, the character in the book could not be anybody else BUT them, then you needed permission.
Note that this is specifically NOT some iron-clad proof beyond a reasonable doubt, civil cases are proved by lower standards of proof such as "the preponderance of the evidence" (which essentially means that the jury agrees it was more likely than not that something occurred in a certain way).
That would be particularly easy for a juror to decide if the actions taken by the character are unusual, and even if fictionalized and changed up, seem very similar to something the real person did. It does you no good to change minor facts or dates or locations, the jury is just average people, and average people can see through that kind of smokescreen. If the real person themself can recognize the similarity, then they could explain that to a jury in great detail and the jury would likely see it too.
Trying to base characters on real people is a shortcut to avoid having to do the work of imagining original characters. Because it is a shortcut, there are bound to be enough similarities that you will almost certainly get "caught", whether or not the real people decide to sue you for damages to their reputation or privacy.