It depends on the publishing platform, your writing tools, and established conventions. As mentioned in the previous answer, commands and button names are often highlighted using bold type. But if the text might be spoken, or read in something that doesn't support formatting (Notepad, for example), then typography doesn't work as a distinguishing sign.
Similarly, if you start bolding UI elements in instructional text, you can end up with text that's hard to read because it's littered with bold, like a black-and-white checkered floor.
I avoid quotes to identify regular commands and menu items. Instructional text can become littered with quote marks. Instead, I reserve quotes for long titles of options and other UI elements, like this:
Select the "Don't save on exit" option, and then click Save.
You can find style guides produced by Apple and Microsoft. Both provide abundant advice about labels for UI elements, typography, procedural language, and other concerns of technical writing.
Trivia: For years, Apple called one type of interface element a "radio button" while Microsoft called the same thing an "option button." One required you to write "check box" (two words) and the other used "checkbox" (one word). I used to have all of this memorized, and then mobile interfaces and the Web happened, and things become less formal.
The important point is to be consistent. In most cases, you should also provide a "key to typographic conventions" in the documentation, where you explain the use of bold, italic, and other typographic or language conventions. Look at any long manual for an example of this.
