In my book I've made up a card game where I've already thought out the rules. Now I want some of the characters playing this game.
My questions are:
For example, is there any point where a character "bluffs" and the reader must understand the rules to recognize that he is making the "wrong" play?
My advice is to consider how the game "fits" into the story, and treat it as a meta narrative device presented in the way it is meant to be perceived. If it is a confusing game with lots of rules, describe it in a confusing way with disconnected rules and anecdotes. Make it confusing to the reader so they understand the game is confusing in-world. It's a kind of show-don't-tell.
If it is a game of arbitrary luck, present it as a toss-up with no skill involved – even if that means exaggerating this aspect, or presenting the game in a different way to how the characters actually play it. Individual characters may see their luck differently, hopefully this tells us something about their character arc and informs us about their worldview.
If it is a game of intense strategy, the deep calculation of one or two "hands" can set an example without boring or confusing readers with every rule. You can specifically show the same hand, so the reader has seen it before and understands the stakes of that particular strategy.
In worldbuilding, less is usually more. The goal is to make your environment feel "natural", not to show how inventive and clever you are with one particular element which often backfires if it's non-essential. If the game is simple, readers should grow an idea of the rules and obstacles just through casual conversation.
Of course you can have one character teach the game to another, but again the focus should be on their character arcs and all the subtle interactions revealing more about them than the actual game.
Stories aren't about connecting all the dots – there's magic that happens in the reader's head when they connect the dots for themselves, that's why infodumps should be avoided. Basically nothing should be in the story that doesn't contribute to the theme, mood, or character arcs.
An in-world game with specific rules sounds like a fun idea, but the reader should be focused on the characters who are playing the game. Whether readers would be able to win the game themselves is probably not essential to the story.
The question largely depends on whether the game and its particular rules are important to the story or not.
Take, for example, Quidditch, from the Harry Potter series. The game constitutes a major story element in the first six books, and the key to a plot coupon in the seventh. So much story-time is devoted to it, that the readers need to understand what's going on. Since the rules need to be explained, they are explained within the story, when a character teaches the MC how to play.
A different example: 3D chess, in Star Trek, the Original Series. Spock is seen playing more than once - against, Kirk, against a computer etc. But it's not a major story element. It merely serves to illustrate that Spock is smart, and that in the age of space people play chess in 3D (which is a nice worldbuilding element - chess was made to simulate battle, and in that setting battles would be fought in space - in 3D). The exact rules of the game are unimportant. Since they're not important, they're never explained (and in fact never got invented in the first place).
Now, your card game - are the specific rules important to the story? Do elements of the story hinge on the particular rules? Then explain the rules, in-story. Is it that only some particular elements need to be understood (there's bluffing and money changes hands, for example)? Then mention only those elements - like chess in the Star Trek example. Is the particular game not important at all, it's just that you want a made-up game for your made-up world, and it's in fact Poker for all intents and purposes? Then don't mention rules at all, just say your characters are playing [InsertGameNameHere].
As the others have said, the details are not important.
Readers will want to know whether this is a game of skill (chess), of chance (most dice games), or of bluffing (poker). It will probably be all of these, but what is the most important? Characters may disagree. Most beginners believe poker to be a game of chance, while experienced players know that the bluffing aspect is more important.
Another important aspect is if it is possible to cheat. Have another ace up your sleeve? Could always come in handy.
You asked if you should put the rules in an appendix. I would say no, not today. Thirty years ago it would have been a good idea, but today we have the Internet.
Make a commercial web site, with a list of your books and how to get them. Put the rules there too.
I agree with all of the other answers: you should only explain rules if it helps telling the story.
But I would like to add a suggestion: if you ever happen to explain the rules in-story, you could explain them in un unreliable manner, from your characters' POV. Some examples include:
a young boy, who is struggling to survive in a tough world, learns the rules by himself as he goes, and loses a lot of money in the beginning because he never got how the "ace wins all" rule works, then he learns the right way and starts winning;
an experienced player may know of a tricky strategy, and explains it (and only that) to another character;
a crook introduces his victim to the game, but doesn't tell about a specific rule that allows him to win;
a stranger plays with a group of locals, and an argument arises between them because in the stranger's land certain rules differs from those the locals are used to;
you might even have your character always win because he "knows how to handle the red women", which is a mysterious sentence to describe some strategy that revolves around the queens of hearts and diamonds, but he never explains what that actually is. It may even be nothing but a bluff, something like "the secret weapon is believing in yourself".
You may not even have to explain the other rules, only enough to show a peculiarity of one or more characters.
Unless understanding the rules is crucial to the story it is probably best not to try to explain them comprehensively in one big chunk as this sort of info-dump tends no to make very entertaining reading.
Of course it's not impossible to do this, having a conversation where one character is explaining the rules to another could work but you should probably only do this if it adds to the story in itself not just to get the rules in somewhere because you have gone to the trouble of making them up.
Having said that explaining the rules of a game to somebody without actually being able to play it tends to be a frustrating exercise in really life and may not add much to a story.
What you certainly should do is make sure that where the games does figure in the story it is consistent with the rules you have devised. It if the outcome of a game affects events it if probably worth giving the reader some idea of what general category of game it is.
There are a couple of examples of important games in Terry Pratchett's Discworld series.
The first is Cripple Mr Onion, whcih is a bit like poker. This is mentioned in passing in a few books but is central to one scene in Witches Abroad which is quiet a key early moment in the development of the Character of Granny Weatherwax who ends up being one of the big pivotal characters in the whole series and one of the most loved.
The other is Thud when again gets several mentions in passing but is a key thematic element in the book of the same name. This is a board game based on a fairly obscure Anglo Saxon game not entirely unlike Chess or Draughts.
In both cases you get enough information in passing to have a fairly good idea of the overall character of the game but not comprehensive rules, although both have been subsequently deduced and now exist in playable form.
It would probably be ok to put the rules in an appendix as an item of interest but you don't really want to create a situation where you are forcing the reader to do 'homework' in order to understand the story.
Iain M Banks "The Player Of Games" is the gold standard for me on this question. The game and its rules are the focus of the whole story, but only as much info as is needed to follow the plot is given to the reader.
