13.5 Reliability in Unobtrusive Research

This final section of the chapter investigates a few particulars related to reliability in unobtrusive research projects, especially as it relates to stability, reproducibility, and accuracy that warrant our attention. These particulars have to do with how and by whom the coding of data occurs.

Stability refers to the extent to which the results of coding vary across different time periods. If stability is a problem, it will reveal itself when the same person codes the same content at different times and comes up with different results. Coding is said to be stable when the same content has been coded multiple times by the same person with the same result each time. If you discover problems of instability in your coding procedures, it is possible that your coding rules are ambiguous and need to be clarified. Ambiguities in the text itself might also contribute to problems of stability. While you cannot alter your original textual data sources, simply being aware of possible ambiguities in the data as you code may help reduce the likelihood of problems with stability. It is also possible that problems with stability may result from a simple coding error, such as inadvertently jotting a 1 instead of a 10 on your code sheet.

Reproducibility, sometimes referred to as intercoder reliability (Lombard, Snyder-Duch, & Campanella Bracken, 2010), is the extent to which one’s coding procedures will result in the same results when the same text is coded by different people. Cognitive differences among the individuals coding data may result in problems with reproducibility, as could ambiguous coding instructions. Random coding errors might also cause problems. One way of overcoming problems of reproducibility is to have coders code together, at the same time.

Finally, accuracy refers to the extent to which one’s coding procedures correspond to some pre-existing standard. This presumes that a standard coding strategy has already been established for whatever text you are analyzing. It may not be the case that official standards have been set; however, perusing the prior literature for the collective wisdom on coding in your particular area is time well spent. Scholarship focused on similar data or coding procedures will no doubt help you to clarify and improve your own coding procedures.