Same Data, Different Conclusions

Schweinsberg, M., Feldman, M., Staub, N., van den Akker, O. R., van Aert, R. C. M., van Assen, M. A. L. M., Liu, Y., Althoff, T., Heer, J., Kale, A., Mohamed, Z., Amireh, H., Venkatesh Prasad, V., Bernstein, A., Robinson, E., Snellman, K., Amy Sommer, S., Otner, S. M. G., Robinson, D., . . . Uhlmann, E. L. (2021). Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis. Organizational Behavior and Human Decision Processes, 165(July), 228-249. https://doi.org/https://doi.org/10.1016/j.obhdp.2021.02.003

Abstract

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects  of  scientists’ gender  and  professional  status  on  verbosity  during  group  meetings.  Not  only  the  analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.

Media Coverage

💡
Chinese Journal of Social Sciences
Strengthening public trust in science
💡
Horizons - The Swiss Research Magazine
Same data, different conclusions

Additional Resources

DataExplained
DataExplained was developed for a recent crowdsourcing science collaboration and helps scientists and data analysts explain their analytical choices more transparently. DataExplained can help justify both preferred and rejected analytic paths in real time. Users can obtain a graphical representation of their workflow to help them communicate their analytical choices to others.

Subscribe to Martin's newsletter

jamie@example.com
Subscribe