You cannot be sued for slander, that is another category of tort and doesn't involve the written word. If you intend to use a real location, (a specific location as the question suggests, one that can be pinpointed down to an address), you CAN be sued for libel, but again, libel is also intended to protect people, their business and reputation; less so places. If you keep specific, identifiable information out of the novel it is unlikely that a libel suit would follow, even if the owner and neighbors can tell which property is being discussed in the story; the problem and liability comes in when the PUBLIC can ID the property based on your story.
If the book is clearly labeled as "FICTION," then the answer, though still yes, carries two caveats:
A judge will most likely toss it if the complainant is suing based on printed prose regarding a place as opposed to a person or business. Especially if there is no way for a jurist to logically connect a specific address with the story.
Even using the name of a small town, street or community is not specific enough for libel. Would be best to create a fake address (if needed) and modify the description of the property. If you are compelled to be SO specific about a location and description that it could potentially ID the address and the resident/owner--yes they can sue -- a competent judge would look it over (you don't want that)...whether or not they can win is an open question for a jury. If you do in fact plan to publish a real address, you are probably violating the privacy of whomever lives there --> THAT is a civil tort you can easily be sued for libel AND more you'd be more likely to lose.
Note: In the US, the 1st amendment gives us WIDE latitude to say anything we want, but the Supreme Court ruled that we have a right to privacy. "My rights end where yours begin," so If I violate YOUR privacy, I've over-extended my right to print whatever I want. In this instance, not a crime, but certainly a good civil-action attorney could prove physical, emotional or financial damage.