A good SDK (software development kit) includes plenty of well-documented examples. It also includes good tutorials and developer guides, which introduce concepts in logical progressions, typically showing only the relevant excerpts from the sample code. (Nobody wants to see a 200-line program inline in a book, but it's important to show that 20-line excerpt that demonstrates a principle right there in the section that talks about that principle.)

This creates a maintenance challenge: over a progression of releases, interfaces or preferred coding patterns change. An IDE (a code-development environment) provides tools for finding the example programs that need to be updated (e.g. find all places where this function is called), but they don't tend to help with references in documentation. So what usually happens, in my experience, is that before a release somebody will page through the documentation looking for suspicious code snippets. This is, obviously, not 100% reliable. (Edit: the full examples are tested regularly, but that doesn't guarantee that the excerpts in the documentation remain in sync with the full example.)

Currently we rely on the technical writers, who we hope remember which examples were excepted in documentation where, to react when the example code or the relevant interfaces change. The team is fluent with the bug-tracking and source-control systems, including subscribing to check-ins, but we are still relying on people's personal knowledge, which becomes a problem if people leave the team. (As is often the case, testing of documentation tends to be a low priority for QA.)

I am looking for practices that have been used successfully to improve the ongoing accuracy of code examples in documentation.

  • 1
    RE your recent edit: You should definitely be extracting your excerpts from the full tested source each time you generate your documentation. Obviously doing this by hand doesn't solve the maintenance issues, but it's easy to automate. Check out the additions I made to my answer last night. Mar 23, 2012 at 15:44

7 Answers 7


Python has a useful module called doctest. It is commonly used to validate tutorial documentation and examples embedded as comments in the code.

The doctest module searches for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work exactly as shown. There are several common ways to use doctest:


To write tutorial documentation for a package, liberally illustrated with input-output examples. Depending on whether the examples or the expository text are emphasized, this has the flavor of “literate testing” or “executable documentation”.

Since the code you're interested in is merely excerpts from the full source, you would extract the excerpts from the tested code based on some metadata. Tools like Doxygen are intended for this purpose.

A language-agnostic approach would be to include the full source for each excerpt in the library source code or wherever the primary documentation resides. Then when you want to build your tutorial/developer guide, you run automated tests on the code using your xUnit-equivlent with doctest-like connector code if necessary, and then extract the excerpts with Doxygen.

Regardless of what specific solution you implement, best practice for maintenance is to follow the DRY principle. Do what you can to keep all of the source code in one place. In your case, it sounds like this will require generating your excerpts from the original sample code each time you generate the documentation. There's some discussion on the topic of code sample testing and maintenance in The Pragmatic Programmer on pages 26-29 (DRY principle) and further on pages 100-101. The authors describe vaguely how they accomplished what you need:

[...] using the DRY principle we didn't want to copy and paste lines of code from the tested programs in the book. That would have meant that the code was duplicated, virtually guaranteeing that we'd forget to update an example when the corresponding program was changed. For some examples, we also didn't want to bore you with the framework needed to make our example compile and run. We turned to Perl. A relatively simple script is invoked when we format the book -- it extracts a named segment from a source file, does syntax highlighting, and converts the result into the typesetting language we use.

  • 1
    Doctest is a brilliant idea. Simply compiling your example code will tell you if it is legal. Paying attention to warnings will tell you if it uses any deprecated features (hopefully). Testing it to see if it produces the expected output will tell you if it does what it says it does. Running style checkers, bug finders, and other linting tools over it will tell you if sets a good example. This doesn't cover every base, but it's an excellent start. Mar 22, 2012 at 22:43

One suggestion is to solve the problem outside of the IDE, and inside your versioning tool.

If you check in your documentation along with your source code to the same repository, then you can use command-line tools like 'git grep' to refactor code + docs at the same time.


ReadTheDocs (a popular system for documenting code-bases) has an interesting feature which may provide an answer to this question: literalinclude.

With this directive, one can include code examples from another file within the documentation. The particularly interesting part is that a subset of lines can be extracted from the source file via the :lines: directive meaning that the source file could be a complete (executable, testable) example while only the snippet that is relevant to the documented section need appear in the docs.

Using this directive, it would theoretically be possible to include all examples in the documentation within the project's test suite. I can't say I've ever gone that far myself - but having run into this precise issue more than once I'm now tempted to give it a try!

  • Oh, very cool! It'd be even better if there were a way to put markers in the file (instead of line refs external to it), so that when somebody adds an include or some code comments (hey it could happen) or the like, it doesn't throw off the excerpting. Mar 17, 2015 at 19:23
  • That would certainly be the ideal. I can see why they did it this way (works with all languages, and sphinx is meant to be language agnostic these days), but it does have that drawback
    – Dave Jones
    Mar 17, 2015 at 23:16

I would actually answer this with my graphic designer hat on: all code should be given a particular style in the layout program (font in particular, but type size, margins, justification), and then you just Search for each iteration of that style. It's still manual, but you won't miss any.

  • The code samples do have their own tag, but you still have to look at each one, find it in the full example, and compare them, which is pretty tedious. (We have dozens of examples producing hundreds of excerpts across a ~1500-page doc set.) I wonder if there's a way to script that, which would presumably require finding a way to annotate the excerpts for what file they came from. Mar 2, 2012 at 14:01
  • 1
    Does the book software you're using have a publish-and-subscribe feature? Mar 2, 2012 at 16:16
  • No, though we can subscribe to source-control changes via email (which is how I make sure I get alerts to code changes I care about). The docs are Docbook (v4.4) XML written in a vareity of editors (personal choice); we use ant targets to do the builds so in principle could check last-mod dates on the relevant examples, but we'd still need a way to say "this programlisting block came from foo.java" in order to check for effects of changing foo.java. (Did that make sense?) Mar 2, 2012 at 17:42
  • ...mostly. I could suggest some strategies for keeping track of owners, but no tested or streamlined solutions. Mar 2, 2012 at 19:08

There are several projects that come to mind that seem to do this well. All of them use very similar techniques. One is the Qt framework, another is a tool called Doxygen, and a third is the GTK+ Project.

In all three cases, the documentation for the project is primarily pulled out of the actual source code for the project. The both are maintained together, often in the same files.

Documentation is stored as marked up comments in the code. Generally speaking, the mark ups can specify everything from the creation of documentation pages, section on a page, cross references, the inclusion of code samples (including full and partial pieces of files), and even images.

Qt and GTK+ both use custom tools to accomplish this.

Doxygen is actually a tool for creating documentation from code, and as such uses itself to document itself.


The best practice would be that all of the code has to be compiled and potentially verified against some code rules.

The way we do this at my work is that all of the examples are tagged in documentation, and then during our nightly builds they all get extracted and compiled. So if someone modifies the interfaces, the nightly build will catch this.

To help us with that we have some custom scripts. Effectively different kinds of examples are marked differently, as they will need different wrapping code to be something that compiles. So we have code that is just a little snippet, complete functions, complete classes, and then some full projects. Each gets compiled, so they are at least up to date with the interfaces/functions.

We currently do not do anything for specific coding conventions, but if you have the first part working, you can then run some automated style checking. Since we do not do this automatically, we do go thru them every so often by hand.

  • We do test the examples in their full form. How do you ensure that the excerpts quoted in documentation stay in sync with the full, tested examples? Mar 23, 2012 at 13:14
  • They are either external files that are merged into the help topics (we use Flare which calls them snippets), or they are marked with a specific CSS style (originally for formatting, now used for code extraction). These are then taken from help pages, wrapped in some standard wrapping code that makes them not be an excerpt any more, and then compiled. Mar 23, 2012 at 15:00

If I remember correctly, The Pragmatic Programmer specifically describes in the book how they built their writing system to make the code extract-able and executable.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.