I am not sure if people would agree, but I think because there were some authors who coined terms and idioms, anyone could create their own idioms. The questions is if there's a guideline, and if there's not what should be common rules our coined idioms should meet. Often idioms are rather simple and don't need any explanation. For example, "like a mad dog" can be understood easily, but what if I coined an idiom like "itching to raise my hind leg", which means "I need to go to the bathroom and take a leak"? Do you think there should be some sort of rules, and how do we go about introducing them in a novel?
Strictly speaking, something is not an idiom until it is in some degree of common use. One can invent a fictional society in which idioms unknown to our current society are in place. But until others adopt them, the kind of thing you mention is simply a figure of speech, in most cases a metaphor.
There are no hard-and-fast rules for coining a figure of speech. Normally you want it to be reasonably clear. If you have to explain it, it loses most of its value. Although it can be a form of characterization. That is, a character might use an obscure metaphor which a second character explains to a third. This shows us something about the first character, and perhaps about the others involved also.
Figures of speech can be used by characters in dialog, or in narrative or descriptive passages. In dialog, they should fit the character and the circumstances, that is, be something that it seems plausible that the character would say in those circumstances. In other passages, there should not be an overlly jarring contra st of style. But these are all judgement calls. In the end if it works in context, it is good.
A successful idiom works for several reasons at the same time.
Often idioms are rather simple and don't need any explanation. For example, "like a mad dog" can be understood easily
This is a good observation. Therefore:
1. An idiom must be transparent (at least when it is initially adopted).
"My day went like clockwork." That is, everything meshed perfectly, as do the gears in a clock, which are perfectly machined to fit together - without any undue friction or hangups.
On the other hand, it's less transparent why you would wish an actor luck by bidding them to "break a leg." One source suggests that's inspired by a superstitious fear that it might be bad luck to explicitly say "Good luck."
2. A good idiom or saying must be well-proportioned to the emotional investment being put into it.
Euphemisms for urinating and/or defecating include "take a leak," "take a dump," "visit the loo," "nature calls," etc. They are usually fairly short. When excusing yourself to "go to the bathroom," you may dismiss it jokingly, but you also don't want to be saying something long and complicated when you're "hopping on one toe."
Compare, not exactly an idiom, but Jesus's phrasing when speaking of those who hurt / corrupt children: "...It were better for him that a millstone were hanged about his neck, and that he were drowned in the depth of the sea." (Matthew 18:6) The subject is not a light and passing one, and the intensity of emotion may support a larger phrase, and a more complex image.
3. The phrase must be sufficiently humorous or evocative to bear repeating.
"Cement overshoes" is used and reused because the associated image is striking and memorable.
But what is or isn't evocative is the point where there isn't a pattern to follow anymore. To suggest someone "take a long walk off a short pier" is a silly and indirect way of saying "go jump in a lake," which is itself an idiom for "go away." On the other hand, "They took him for a long walk" sounds more ominous than humorous. At least, in the mindset brought on by talking about concrete overshoes, it sounds ominous.
There's not a good formula for memorable. But I would propose these three are the main things to make something believable as a fictional idiom, and maybe even capable of becoming a real-world idiom.
On the other hand, "Catch 22". (An arguably meaningless phrase, outside the context of a book I haven't read - but the idea attached is just so useful to be able to say. Because rule 0 is that an idiom must be a useful way to communicate something people want or need to communicate.)
A similar musical concept exists; essentially, Repetition Legitimizes.
The more often you use these new (or even just uncommon) idioms, the more they will blend into the story. If I wrote an entire book in an internet-based slang for dialogue, and then had a character say, "Attend, prithee", the readers would throw my book out the window. Writing a whole story with a character or two using uncommon or nonexistent phrases allows those phrases to sound more natural and less forced.
As others have noted, the phrases you make up should probably be either explained in the book (and you'd better have a good explanation), or be easily understood.
Hemingway’s For Whom the Bell Tolls includes a number of Spanish idioms translated rather directly into English. For someone unfamiliar with the Spanish language and culture they might as well have been coined by the author.
In the TV series Firefly, the characters often utter something in Chinese, leaving most viewers completely in the dark as to the literal meaning of what was said, but from context it’s usually apparent that the character is cursing to express dismay or surprise or something of that nature.
And of course you can have your characters comment on the connotations of any metaphor:
‘You’ll live to regret it, young fellow! Why didn’t you go too? You don’t belong here; you’re no Baggins—you—you’re a Brandybuck!’
‘Did you hear that, Merry? That was an insult, if you like,’ said Frodo as he shut the door on her.
‘It was a compliment,’ said Merry Brandybuck, ‘and so, of course, not true.’
So go for it to serve whatever narrative purposes you have in mind.
Coining a new idiom is easier than blinking twice and seeing daylight, so you should be as relaxed as a cat on a warm sunbed when you innovate new ones.
You can ask your friends which of your idioms fit like old favourite gloves on an autumn's walk in the park and which ones jar like a Nazi armband in a feminist peace march.
If you aren't sure, remember that for every caviar-eater, there's one that prefers bacon, so don't worry too much at the occasional rejection.
If in doubt, showing by doing is always a workable solution.
You can't create idioms.
Idioms are sayings adopted and chosen by the people who use the language over a long period of time.
It is an organic process. No one can create an idiom.
Any attempt to do so will appear transparently phony to virtually everyone.
"Stop trying to make 'fetch' happen! It will NEVER HAPPEN!"
And yes... the new "idiom" candidates above are ALL TERRIBLE, and will never happen.