Which is the best software to write a book which is similar to Fundamentals of Physics by Resnick and Halliday?
I want to self-publish a book (on physics) on Createspace but have no idea regarding the software would be best suitable for the job.
You could use typical desktop publishing tools like Adobe InDesign or Quark; you could use XML publishing tools such as MadCap Flare, or Paligo, or DITA... all of these will suffice - it's a question of your preferred workflow, budget and available learning time to master the workflow enough to get the quality of results you need.
Heck with time and patience enough, Apple's Pages or Microsoft's Word or Publisher could handle this task, albeit true that I'd not envy either you or your eventual printers if you choose the Microsoft route.
LaTeX is primarily a science-writing focussed open-source typesetting tool, and is often included as a plug-in to other tools to ease complex equation layout on specific spreads - though I myself don't personally know anyone who has used LaTeX to set up and produce an entire large document like a textbook, it clearly can be done that way - though given all the references in the documentation to LaTeX not being a layout environment and to letting authors remain authors and leaving the document layout to document designers, one suspects there may be a lack of tools to get top-shelf layout results.
I am (amongst other things) a document designer as well as tech writer - were it me, I'd either use InDesign or Flare - though if Affinity finally completes the development of their new Publisher application (Affinity Photo and Designer are directly out-competing Adobe Photoshop and Illustrator) I'd probably seriously consider that too.
From a quick look at the Createspace pages, they're expecting either print-ready PDFs or Word files... so quite possibly the best answer is to not overthink this: use whatever tool you are already familair with for producing typical .doc files, whether that's Open Office or Pages or Word or Libreoffice or Google Docs.
LaTeX tends to be an industry standard for "mathy/sciency" open works, but you'd have to program some pretty sophisticated layout schemes to get to anything that looks like a high-school math book. LaTeX is a platform, not a software solution. You learn a language and write files using that language and then the platform has the ability to render it in several different ways.
Caution: If you've never done any programming, learning LaTeX to the point of proficiency is about as difficult as taking a year's worth of classes.
Oh, you definitely have to use LaTeX, but there's more than one way to skin that cat.
I recommend a program called LyX. It uses LaTeX under the hood, but it saves you much of the harder coding, and lets you see roughly what the document would look like before you export it to an appropriate format. Its document class choices make it incredibly versatile; I've used It to write books, papers, scripts, you name it.