What is the purpose of the reference? If it is essential to the reader's understanding of the story that he understands the reference, than I'd be more likely to add some explanation. If it's just a side comment to add a little flavor to the story, then probably not.
As What says, you can often easily toss in a couple of words that give the reader at least a general idea. "We went to an exhibition of the works of the surrealist Spanish painter, Salvador Dali." Even if the reader has never heard of Dali, he now has at least a general idea of who he is.
Often you can give clues from context that will make a reference at least generally clear. Like, "At the art museum, Sally was impressed by the works of Salvador Dali." From the fact that we say it was at an art museum, the reader would likely guess that Dali is an artist and not an auto mechanic.
A reference can add something for the person who recognizes it while not seriously subtracting for the person who doesn't. "'This conversation is getting totally surreal!', John cried. 'It's like a painting by Salvador Dali!'" Someone familiar with Dali's paintings would likely find one or two of them floating to his mind and helping to set the mood of the scene. Someone not familiar with Dali would get the idea that he's a surrealist painter and move on. The reference didn't help him frame the scene any, but it didn't really hurt, except to the extent that his enjoyment of the story is interrupted as he puzzles over the unfamiliar reference.
At some point I think you have to make reasonable assumptions about the background of your readers. If you make a reference to George Washington in a book intended for an American audience, I think you can safely assume that your readers know who he is. If you mention the man who was the deputy mayor of Hartford Connecticut in 1824, chances are that even readers very knowledgeable about history do not know who he is.
Two other stray thoughts:
One: An explanation can ruin an allusion. I saw a movie once with a scene where a man is angry at his wife because he felt she did not dress up attractively enough for a social event that could advance his career, and he says, "You looked like Quasimodo!" Then a couple of other angry sentences go by and then she says, "Well, you've just compared me to the Hunchback of Notre Dame, and ..." So okay, I get it, the writer thought that having the man say his wife looked like the Hunchback of Notre Dame would be a good, nasty insult that fit the mood of the scene. But then he was worried that many viewers might not know who Quasimodo was, so he put in the explanation. TO me, it just sounded lame. Obviously the husband knew who he was talking about when he made the reference. Why would his wife have to tell him who he was referring to? It was just so obviously for the benefit of the viewer and not the characters that it was jarring.
Two: An allusion can be an "inside joke" and leave it at that. Like, I once wrote a book in which I made a reference to a variety of religious and philosophical texts, and I wrote "like the Book of Mormon, the Koran, the Communist Manifesto, or the Seldon Plan". The "Seldon Plan" is a book in the science fiction novel "Foundation" by Isaac Asimov. I gave no further explanation. SF fans will get the reference and, presumably, laugh. Those who don't recognize it will likely say to themselves, "whatever, must be some other religious or philosophical book". No one has ever asked me about it, and so I just snicker to myself.