Blog

This lighthearted cartoon (shown in full below) makes an important point about classical hypothesis testing: if you test enough things you’re bound to find something that’s significant.

The way to avoid this is to determine before you gather the data what questions you intend to ask and to avoid going data dredging after the fact. This approach is enshrined in the regulation of Clinical trials, for example, but in the absence of hard-and-fast rules for other areas of statistical application, the cartoon below illustrates the importance of seeking statistical advice to avoid this and many other pitfalls of a naive application of statistical methods.

XKCD “Significant” comic: http://xkcd.com/882/