Problem set 9

Submission information: please submit on ZoneCours

Look at the effect size and power examples; this should be helpful in completing the problem set.

Task 1

The following text is a quote from Investigating variation in replicability: A “Many Labs” Replication Project by Richard A. Klein, Kate A. Ratliff, and Brian A. Nosek (pp. 13-14) about a replication of Oppenheimer & Monin (2009) work on the retrospective gambler fallacy.

The differences between groups was reliable, omnibus \(F(2, 77) = 4.8\), […] Cohen’s \(f = 0.18\). Pairwise comparisons showed that all differences between conditions were reliable as well, \(t(47, 48, 57) = 1.94, 2.32, 2.65\); \(p < .05\), Cohen’s \(d\) = \((.56, .67, .69)\).

  1. Using the information from the quote, compute the effect size for Cohen’s \(f\) and \(\widehat{\omega}^2\).1 Note that you won’t normally get the same value of Cohen’s \(f\) that is reported.
  2. Compute the overall sample size necessary to replicate this study with a power of at least \(0.99:\) compute the sample sizes for both the overall effect (using the value you calculated in 1.) and the three pairwise \(t\)-tests based on the reported values of Cohen’s \(d\) and pick the maximum sample size needed for a balanced design, i.e., gathering the same number of participants in each subgroup.
  3. Read the results of the replication in ManyLabs1; these are illustrated in Figure 1 and summary statistics about standardized mean differences (Cohen’s \(d\)) are reported in Table 2 (Klein et al., 2014) under retrospective gambler fallacy. Comment about the success of the replication.
  4. Based on observation of Figure 1 of Klein et al. (2014), why might one object to using estimated effect size reported in peer-reviewed papers?2

References

Klein, R. A., Ratliff, K. A., Vianello, M., Adams Jr., R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., … Nosek, B. A. (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45, 142–152. https://doi.org/10.1027/1864-9335/a000178
Oppenheimer, D. M., & Monin, B. (2009). The retrospective gambler’s fallacy: Unlikely events, constructing the past, and multiple universes. Judgment and Decision Making, 4, 326–334.

Footnotes

  1. In R, check out the function F_to_omega2 and F_to_f from the effectsize package. You can also use the formula presented in the slides.↩︎

  2. Hint: are the original effect size in line with the replications or not?↩︎