These "Tutorials in Consumer Research" were written by topic experts to provide insight into conceptual and methodological aspects of research. These author perspectives provide researchers with some help and insight on approaches to conducting high-quality research; they have not been peer reviewed by additional experts in the field.




Tom Meyvis

Stijn van Osselaer

Experimental social science, including the field of consumer research, has recently been shaken by what has been termed the "replication crisis." Researchers have become aware that many of our field's findings are difficult to replicate and might, in fact, not be true. The estimates of effect sizes in published work tend to be inflated as a result of the use and abuse of "researcher degrees of freedom," such as selectively omitting studies or conditions, or making selective decisions about the use of covariates and transformations or about the removal of participants from the analysis ( Simmons, Nelson, and Simonsohn 2011). When researchers make these analysis and reporting decisions post hoc, based on the extent to which they increase the apparent strength of the evidence, the published work will overstate the likelihood that the effect can be replicated using the same procedure and analysis. Given that true effect sizes tend to be substantially smaller than those we would infer from published research, this implies that we tend to overestimate the power of our studies.
 
One response to this replication crisis has been to advocate increasing power through the use of larger sample sizes. We wholeheartedly agree that increasing sample sizes is often essential to reduce both false positives and false negatives. However, researchers regularly face practical limitations in the extent to which they can increase their sample size -- for example, when experiments rely on expensive lab equipment, hard-to-find populations, or expensive incentives. Furthermore, even for relatively uncomplicated studies that require only minor incentives, it may not be feasible to recruit the extremely large samples needed to detect small effects with acceptable power. Our argument in this tutorial is that, rather than try to boost power solely by increasing the sample size, researchers should also aim to boost power by increasing the observed effect size ( Sawyer and Ball 1981). They could do this, of course, by investigating only phenomena that commonly produce large effects. However, this is neither desirable nor necessary. Although subtle relationships tend to produce small effects, these relationships can have important consequences. Furthermore, the size of an effect is inherently malleable: appropriate analysis and design decisions can increase the observed effect size such that even somewhat subtle relationships can be studied with feasible sample sizes. We therefore advocate increasing the size of the effect of interest by employing (1) appropriate data cleaning techniques, and (2) better research designs and experimental procedures.
 
Read More

Volume 44, Issue 5, February 2018
DOI: 10.1093/jcr/ucx110