As with any profession --doctor, say, or lawyer --people outside the profession have a set of vague stereotypes about writers, formed by books and movies, and with little-enough basis in fact.
The mythical writer is a novelist. He is a man. He is at work on his next bestseller, but is afflicted by writer's block. He is intelligent and intense, but perhaps a bit pompous and self-important. He lives in the world of his book. He writes all the time, and everything he writes gets published immediately. He wants nothing more than to be given ideas for his next book. And so forth, and so on.
And of course, there are ample stereotypes about the look of the writer as well. As it so happens, I found your question puzzling, until I looked at your profile picture. I would venture that, for whatever reason, you look like many people's internal stereotype of what a writer should look like. I imagine if you humored one of these people by saying "yes!" they would lean in close, with a look of satisfaction, and say "I knew it!"
But don't expect them to actually be interested in anything you may have written.