As a writer and a (former) typesetter/graphic artist/production manager I have to state that writing a book =/= designing a book. You can't write your book in Word or TeX and have it be sufficient for the printed element. (That part is opinion but based on experience. Dedicated hardcore users of TeX might disagree with me.)
If you're writing a book, just write the book. There are plenty of posts on Writers on how to do that. But I would highly recommend using a program designed for book building like Scrivener. That will allow you to organize chapters and sections and allow you to import formulas from TeX so long as they've been exported as images (for reference only, NOT composition.). All you need to do then is label the formula and make a reference to them in your work ("In Diagram 2.11, the Pythagorean theorem...").
If you sell your book to a publisher they will compose the book using their preferred methods. But you will want to make sure you have all of the formulas saved in Tex (or other formula able software).
But if this is a project you intend to write, design and publish all by yourself then you should really invest (and I do mean invest) in a proper pagination program like Adobe InDesign or Quark Xpress.
Regarding scientific and mathematical formulas, InDesign can import formulas directly from [shudder] Microsoft Word's equation editior. InDesign can also use formulas exported from Tex as images and there is also a plugin called MathTools that can create formulas right in InDesign. Similarly, Quark can import TeX formulas as images and there is a plugin (Xtension) called Math+Magic that will more easily import formulas from TeX.
The caveat is that even if you have the right tools, you still need the right training to make them work. That is why I strongly suggest that you keep your work very organized so that if and when you decide to outsource the design and pagination of your book, you can.
You might also want to check out the Tex and Graphic Design forums on Stack Exchange to see how other users tackle this problem.