lornecampbell

science and psychology

Deciding What Studies to Replicate: The Path of Least Resistance

3 Comments Posted by on May 5, 2015

I support the efforts of researchers in our field to conduct direct/close replication attempts of published research. It is something I do myself: https://lornecampbell.wordpress.com/2015/03/18/confessions-of-a-replicator/. It is wonderful to see journal policies starting to embrace replication studies, including special issues in different journals devoted to replication studies in a given area of study. But…not really a but in the sense that I will now soften my support for replication research, but a but in the sense that because less labor/time/cost intensive studies will likely be the focus of the majority of these replication attempts compared to more labor/time/cost intensive studies it seems inevitable that an asymmetry in the precision of effect size estimates will develop over time.

Imagine a researcher that has decided to devote some research effort to replication studies comes across three published studies that pique his interest:

  • Publication 1: recruited a large number of participants from either a University subject pool (i.e., undergraduate students that need to participate in research studies for course credit) or online (e.g., Amazon’s Mechanical Turk).
  • Publication 2: recruited undergraduate students from the University subject pool, but had each participant come into the lab individually because of complex experimental manipulations as well as the collection of biological samples.
  • Publication 3: recruited participants from the community, following participants over a period of two years with multiple testing sessions (both in lab and online).

I do not need to provide more details for these fictional studies to make the point that the labor, time and cost needed to directly replicate the methods of study 3 is much greater than for the other two studies, and is greater for study 2 compared to study 1. Given that researchers do not have access to unlimited resources over a prolonged period of time to conduct their own research let alone direct replications of the research of others (if you do, call me), it is reasonable to conclude that of the fictional studies presented more replication attempts would be made for study 1 than the other two studies. Over time, therefore, more precise estimates of the effect sizes obtained in “easy to run” studies will accumulate compared to “difficult to run” studies. Put another way, one shot correlational and experimental studies involving University students or MTurkers will be the focus of the bulk of replication attempts; studies with special populations (e.g., cross-cultural samples, married couples, parent-child interactions, and many, many others), those collecting “expensive” data (e.g., brain scans, hormonal assays), and studies using longitudinal designs (e.g., daily-diary studies, the early years of marriage, personality development across time, and so on) will be the focus of few, if any in some cases, direct replication attempts. I cannot imagine, for example, obtaining the grant funds necessary to directly replicate a multi-wave study of newly married couples over a period of two or more years [but see comment below–Brent Roberts did receive grant funding along these lines]. Even if funds were on hand to directly replicate a two-week diary study that included pre- and post-diary assessments, the amount of time needed to run the study, and the research assistants needed, would likely dissuade most researchers from endeavouring to replicate this research.

Now that the value of direct/close replication studies is generally recognized, perhaps we need to find ways of incentivizing replication attempts of studies that otherwise would be ignored by most replicators.

Uncategorized

3 responses to “Deciding What Studies to Replicate: The Path of Least Resistance”

  1. bwrobertspubs May 5, 2015 at 4:24 pm

    Hi Lorne, no need to imagine that a granting agency would fund a multi-wave longitudinal replication structure because NIA already did so for my studies. These 3 longitudinal studies are not direct replications of each other, but they are close enough in structure to confirm findings across data sets. Moreover, we just published a close replication of a multi-wave longitudinal study with another multi-wave longitudinal study in JPSP. Yes, even longitudinal research can be directly replicated.

    So, while I agree with the sentiment of your post, I think it all boils down to how much we care about a phenomenon. If the topic matters enough, we can and should do the work to show that the phenomenon is robust.

    Liked by 1 person

    Reply
  2. David Schmitt May 5, 2015 at 5:38 pm

    UNLESS someone like you

    cares a whole awful lot,

    nothing is going to get better.

    It’s not.

    Like

    Reply
  3. Sanjay Srivastava May 5, 2015 at 8:09 pm

    Great post Lorne, and an important issue.

    To some extent we have a similar dilemma in original research – it’s a lot harder to do Study 3 the first time around too. People will debate how well we’ve addressed that problem. There are plenty of think pieces lamenting the lack of diverse samples, non-self-report methods, and “real behavior” in psychology research. (FWIW there are also think pieces lamenting the opposite e.g. that we are sometimes taken in by flashy and expensive methods like fMRI when they don’t actually add much value). To whatever extent we have dealt with those problems, it’s because editors, reviewers, and the field at large value the the methods’ scientific suitability over their convenience. If that translates into funding and publications, it creates an incentive for researchers to take on more challenging projects.

    With replications, for the last couple of years a lot of people are just applauding anybody who publishes direct replications, because it is still relatively rare. But as it becomes more common (fingers crossed), you are absolutely right that we need to start thinking about how we value replications. Just as with original work, there are probably bad criteria as well as good. For example, we should be careful about assigning value on results — we don’t want to value a “failed” replication more than a “successful” one (or vice versa). But if a replication involves an important question that many are reluctant to tackle because of the methods, hopefully we’ll proportionally reward the few who do.

    Liked by 2 people

    Reply

Leave a comment