How do occupational psychology lab experiments hold up in the real world?

A new paper combs meta-analytic data from across psychology to better understand how laboratory research translates to the real world. Here at the Occupational Digest, we report on evidence from a range of sources, often field work from within specific organisations (such as managers ratings of performance), or more general surveys of working individuals - but also from laboratory studies, typically involving students rather than participants with a working history. Are findings from such set-ups likely to be reliable?

The answer from Gregory Mitchell's analysis is a qualified yes. He looked at 217 lab-field comparisons drawn from 82 previous meta-analysis that covered the breadth of psychological research areas, looking at when a finding in the lab was corroborated in in the field. This follows an earlier analysis using a smaller data-set by Anderson, Lindsay and Bushman in 1999. At the highest level, Mitchell's analysis dovetailed with the Anderson et al. finding: lab and field reports correlate at an r of around .7, which means a strong relationship between the two. Although this is reassuring, the analysis also revealed that 14% of laboratory results actually changed signs when tested in field; that is, the effect ran the other way to the original finding.

Things get really interesting for the analysis by sub-field. Mitchell drew out the two fields with the most lab-field comparisons available, which were social psychology and, fortunately for us, occupational psychology. Social psychology showed a lower correlation between lab and field of .53. The occupational results, meanwhile, showed a very strong correspondence, with an r of .89. (1 would be perfect correspondence.) Only two of 71 lab effects changed signs for occupational research, with the likelihood of this happening nearly tenfold greater in social psychology. The data suggests that one reason for this is a much higher proportion of small effect sizes reported in the social psych laboratory literature; a smaller effect can much more feasibly 'flip' than a large one, all else being equal.

All told, the use of laboratory studies to inform workplace psychology appears to be in fairly good health. That said, we can't jump from this general finding to the specific confidence that a given laboratory effect will translate into the field with the same strength or even the same direction. We should continue to look for real-world investigations to follow on the heels of laboratory proof-in-concepts. What we can be comfortable with is that this strategy seems to be working pretty well in our field.

Mitchell, G. (2012). Revisiting Truth or Triviality: The External Validity of Research in the Psychological Laboratory Perspectives on Psychological Science, 7 (2), 109-117 DOI: 10.1177/1745691611432343