I am writing a fictional novel. It is set in many fictional places and the book will have it's own set of rules of physics. However, I feel that if I lose the sense of realism too much there will be no relation between the book and the reader and I will lose their attention. Will I still get a sense of relation and if not, can I get it through another method?
Realism has several components. Different ones dominate in different genres/settings and among individual readers.
(Real) setting accuracy: If you're describing a real place or a time in history, people who know something about that will respond based on how closely you match what they know. If there isn't a lot of noise and traffic on your mid-day NYC streets (and there isn't a plot element that explains that), for example, people aren't going to buy that.
Setting plausibility: This is the case you've identified. Basic errors of credibility -- e.g. routine snowstorms in terran tropical climes -- will be noticed by everybody and bother some. Others will be less obvious to some -- e.g. giant creatures that disregard the square-cube law -- but will be just as disruptive as the basic ones to the people who know. This also applies to consequences of plot points; people with medical knowledge may know that your pandemic isn't credible, scientists may know that your meteor strike wouldn't produce the effect you describe, and so on. You need to decide how important this is to your audience.
Behavior plausibility: Unless you do a lot of stage-setting to the contrary, people will generally expect your characters to act like people. Your readers have expectations about how "Joe Average Person" would behave in a given situation; that expectation may be reasonable or unreasonable, or uninformed because your characters are different, but this is what they'll be looking for. Seemingly-rational characters behaving irrationally, experts who don't seem to know the basics of their fields, and so on disturb some readers.
"Magic" plausibility: People are generally willing to go along with a fantastical setting if they think it's internally consistent. The sorcery in your story can be completely unrealistic (from our perspective) but realistic within your story. You do that by exposing the important rules of your world (ideally through illustration, not long exposition) and then staying consistent with that.
I think you will still have a sense of realism. As long as you explain the physics laws/magic laws/whatever differs before they take effect, the reader will know why/how things are happening. As long as your definitions are clear, detailed, and consistent with the effects, you should be fine.
Consider: Any novel that deals with magic has an unknown set of rules. Once those rules are explained to the reader, however, and if the magic follows those rules, things seem perfectly realistic within the novel.
The term you want is verisimilitude.
Basically, you want to avoid breaking the readers suspension of disbelief. This means that what happens must be consistent to the rules of what can happen that the reader has accepted for the setting. If the story is set in the real world, this is close, but not identical, to realism. You'd still be expected to adhere to genre conventions even if the story is set in the real world.
Even in fictional world the genre matters, for example in science fiction you are expected to warn the readers of the changed parts in advance, in fantasy having the setting consistent with itself is sufficient. In fact, regardless of genre the setting must be self-consistent, but usually it is not necessary to make specific effort because your own experience of how the real world works helps you.
In my experience, whether a relation between the reader and the characters of a story can be established depends not so much on the accurate depiction of physics or any kind of back-ground but on the realistic psychological depiction of the character. Can I identify myself with the character? Can I understand what he is doing? (This is very different from liking the character.) Previous answers pointed out this issue, and I'd merely like to comment on it by listing a few examples that I thoroughly enjoyed and that appeared, in a psychological sense, realistic to me:
- The Time Traveller's Wife by Audrey Niffenegger: She muddled up both the laws of physics and biology and wrote, in my opinion, the best love story of the past 20 years.
- Jonathan Strange an Mr Norrell by Susanna Clarke
- The Age of Miracles by Karen Thompson Walker: A coming-of-age story set in a world with severly disturbed boundary conditions. A great book that I couldn't put away, because it was beautifully written and psychologically accurate.
- Do I really have to mention it? A Song of Ice and Fire by George R. R. Martin. Some say Martin revolutionized Fantasy. I tend to agree, because he did the one thing I always missed in Fantasy literature: he gave psychological depth to his characters.
Personally, I recently wrote a novel about transsexuality that uses fantasy elements to explain the transsexuality of the main character (body switch theme). Unfortunately, I have some difficulties to explain to the publishers that this approach is valid, because the psychology of the transsexuality problem is not affected by slightly disturbed physical laws. In my opinion, this is exactly what fantasy should do: Take a conflict out of our world and place it in a situation that doesn't allow the reader to ignore it any longer. In this respect, Fantasy can be immensely powerful, so: Go ahead, don't be afraid to lose touch with reality. I think, as long as your characters are credible, lacking realism will not be a problem for stories set in a fantastical world.
Internal consistency is the key. Build your world and how it works and play by your own rules.... I remember a book which had two types of magicians whose who worked with magic and those that worked with the rules that magic followed... a little weird but it worked because when the way magic worked changed you knew why and how, and in this case it was all part of the plot.