It isn't a "bad idea" per say, as long as there are in-world reasons for what you have contained in it.
It's very common to have fantasy stories based in other cultures. Eon, for example, is based in large part in Chinese mythology. Codex Alera by Jim Butcher is based in Roman culture, even though it is not based explicitly in Rome.
World building isn't always making an entire planet. World building can be as localized as creating the culture of a country, state, town, or even family, so whether world building is strong or weak isn't based entirely on what you get your inspiration from. If you make your world believable, the world building is strong regardless of whether it's based on an earth culture.
For crafting a fantasy novel world, here is a couple of useful links: Fantasy Worldbuilding Questions and Magical World Builder. The questions go through everything, from what the world is based off of to the diet of the people in the world. The other one is a pdf of a 30-day exercise to create a world. It starts with the basics and goes on through building up and then refining a fantasy word.
In regards to combining different cultures, it can be a mistake. While it may seem like it makes sense to take different things from Earth today because they all exist now, at the same time, consider that that is only true because our history progressed as it did. Latin is used for animals now because it was a major western language and the western cultures were, and still in many cases are, the predominant cultures in the world. A lot of international things are based in western ideas or western languages because they spread internationally more than the east or Africa did.
In addition, many cultural ideas or societal norms exist now because of religious values or historical religious values. If you're combining Japan and Rome, for instance, you would end up with major issues in terms of religious ideas and practices. They worship or worshiped their gods differently. Even things that don't seem to be related to religion on first sight are related. Views on homosexuality, for example, are based strongly on religious values. The Japanese historically had no issue with homosexuality, while the Abrahamic religions do.
Language is another issue to consider. Japanese is a very different language from Latin, and if you have both of them, you should have a good reason for it. Someone with a Japanese name wouldn't be using Latin, probably unless they were an academic or a biologist, and that would only be fairly recent. The languages are structured very differently and would likely not exist in the same civilization for no apparent reason.
I was using Latin and Japanese as the two examples because those were the two you mentioned. That is true for any set of cultures. One major thing is that the farther apart the cultures are, the more difficult it will be to combine them to make something cohesive and sensible. If you do want to combine cultures, you will need to decide why they're together (for example, why people in a Japan-esque world are speaking Latin, which is from a completely different language family) and if it's necessary for your story. The other thing you can do is try to take bits and pieces from different places, but you still need to be careful doing that so it's coherent and cohesive and doesn't look like you made a random jumble of cultures.