Illustrated books for adults are not uncommon. Consider for example the illustrated Lord of the Rings or Stardust. In fact, illustrations might serve as an incentive for fans to buy an extra copy of a book they already own.
Here's something for you to consider, however:
Printing a book costs money. Printing an illustrated book costs more money. Considerably more money if you want coloured illustrations, but there are issues even with black-and-white ones (more paper and ink required, more work has to go into page layout etc.) All this money is an investment put by the publisher, in the expectations that the sales would cover it. If you insist on illustration, you increase this original investment, that is increase the risk a publishing house takes by gambling on you. So, you are effectively reducing your chances of getting published.
If that is not enough, there is the whole e-book market: not all platforms take well to illustrations. Phone apps in particular would sometimes skip over the illustrations, and other times not shrink them properly, so they are too big to see. And then there are the translations, which would have to do the same math all over again.
(If you are self-publishing, the risk is yours rather than a third party's, but the considerations do not change.)
For this reason, illustrated books that are not children's books are always books by already established authors, and often (though not always) special illustrated editions of books that have sold well in the past. In such cases, there is little risk for the publishing house - they can reasonably expect the illustrated book to sell well.
As for preconceived images, if some people state this is a problem for them, then for some potential readers it is a problem. I personally am a collector of illustrated special editions, never had any problem with simultaneously holding my own image of whatever there is in the book, and enjoying someone else's images. Never needed a picture to visualise things. From the different answers you're getting to this issue from different beta readers, you can see it might be a better idea to keep the illustrations for the "special edition". That way, you can keep both the illustration-lovers and the illustration-haters satisfied.