Why replication matters: Insights from replicating 10 online privacy studies

Several times, I tried to get a replication project funded. Every time, reviews came back with a the same results: a worthwhile idea, but not feasible. Often the concern were the costs, other times the sample size we would need. Overall, between the lines, there seemed to be a vague doubt about whether replication is really “original” enough to even deserve a grant. Well, times have changed now, but for the longest time, replication was (and probably still is) something our field admires in principle, but avoids in practice. In communication science, direct replications make up only around 1.8% of published work (Keating & Totzkay, 2019). This is what happens when a thing is hard to fund, hard to publish, and slow to be cited.

So out of frustration, at some point, I stopped trying to get funds (Update: I finally managed to get a many-labs project funded!!!). Instead, I thought, let me show you that it is possible anyway. I had the idea to built a consecutive replication project inside a Master’s course at the VU. Over four years, each year’s group of students systematically replicated up to three of ten influential online privacy studies. These were all fielded through the same recruitment platform (CloudResearch) and all powered to detect the smallest effect size the original study had reported. In practice, this usually meant samples of more than N = 780 participants. We always preregistered the design, we reused the original measures, and we evaluated each study against the replication criteria proposed by LeBel et al. (2018): whether the replicated effect sizes were similar in direction and significance, as well as whether the original effect size fell inside our replication confidence interval.

The studies’ range was large and included several key privacy frameworks: the privacy calculus (Krasnova et al., 2010; Hallam & Zanella, 2017), context collapse (Vitak, 2012), the privacy paradox (Dienlin & Trepte, 2015), disclosure management (Masur & Scharkow, 2016), privacy literacy (Park, 2013), privacy tool use (Litt, 2013), antecedents of self-disclosure (Zlatolas et al., 2017), and privacy fatigue and cynicism (Choi et al., 2018; Lutz et al., 2020). The project was very successful so far. The first three replications are now published in the Journal of Communication (Link; Masur & Ranzini, 2025), and the two privacy cynicism studies have now been published as a chapter in the Edward Elgar volume on privacy cynicism, apathy, and resignation (Link; Masur, 2026).

In this blogpost, I will not zoom in on specific findings related to the individual studies but rather describe the bigger picture that emerges when you look at all of them at once. I already presented this bigger picture at the Media Psychology Conference 2025 in Duisburg, but I thought I could share it here as well.

The big picture: Most findings survive

The overarching takeaway is quite reassuring, but perhaps also humbling. Cross-sectional, survey-based privacy findings broadly replicate. When we plot every original effect against its replication, the resulting scatterplot tracks the diagonal of perfect replication reasonably well, and most paths stay statistically significant (see Figure 1). Privacy research thus overall replicates. At least, if we look at the amount of individual paths that replicated in direction and significance. Yet, the devil lies in the details (read on for the surprises!).

**Figure 1.** Every path from every study, original effect size on the x-axis against replication effect size on the y-axis. The bulk of the points cluster around the diagonal and remain significant (orange), with a minority crossing into non-significance (blue).

One insights is that the effect sizes are slightly smaller on average and more wide-spread in the replications (see Figure 2). That shift should be familiar to anyone who has followed the reproducibility work in psychology (e.g., Open Science Collaboration, 2015). Even when an effect is real, the original literature tends to overestimate its size. An explanation might be that first significant results are the lucky ones, and luck does not (necessarily) replicate. So the honest summary is: the direction of our findings is mostly trustworthy, the exact effect sizes perhaps rather less so.

**Figure 2.** Distribution of effect sizes across the whole project. The original studies (left) and the replications (right) have broadly similar shapes, but the average effect is slightly smaller.

Not every finding is equally replicable

Averages hide the more interesting story, which is how much the studies differ from one another. At one end, some models mapped onto the new data almost flawlessly (e.g., Choi et al., 2018; Lutz et al., 2020; Hallam & Zanella, 2017). At the other, a few core assumptions turned out to be far more volatile (e.g., Dienlin & Trepte, 2015; Masur & Scharkow, 2016). When you separate “same direction and significance” from “exact replication” (the original estimate is inside the replication’s confidence interval, see Figure 3), the gap between the two bars is itself the finding: many studies kept most of their paths pointing the right way, while far fewer reproduced the original effect sizes.

**Figure 3.** Proportion of replicated paths per study. Blue marks paths that held in direction and significance; orange marks exact replications, where the original effect size falls within our confidence interval. The blue bar almost always exceeds the orange one.

That said, the overall good picture should not fool ourselves into believing that everything “replicates” (at least by direction and significance). At times, while the overall amount of paths that replicated was high, primary hypotheses (rather than paths e.g., related to control variables) did not replicate (e.g., Dienlin & Trepte, 2015; Zlatolas et al., 2017). Noteworthy, the often contested relationship between privacy concerns and self-disclosure (if non-significant mostly termed the “privacy paradox”) often did not replicate (e.g., Dienlin & Trepte, 2015; Vitak, 2012; Hallam & Zanella, 2017). Other times, key assumptions (e.g., benefits clearly predicting self-disclosure) also did not replicate (e.g., Krasnova et al., 2010). Looking at Figure 4, we do see quite some differences. Particularly if the scatterplots between original and replicated effect sizes widens, it deserves a closer look.

**Figure 4.** The same comparison broken out per study. Some frameworks show a tight, linear correspondence between original and replication; others, with more paths drifting into non-significance (blue), reveal how much dispersion a complex multi-variable model can carry.

What makes a finding replicable? Predictors of replication success

Because we had ten studies rather than just one, we could ask questions that no single replication can answer: For example, what predicts whether a finding survives? Tentatively, three patterns stand out.

First, the statistical power of the original study seems to matter. The lower the original power, the less likely a path was to replicate, in direction and especially in magnitude (see Figure 5). Second, and relatedly, small original effects were the most likely to not replicate. Third, theoretical and measurement parsimony helped: studies built on a few clear mechanisms and previously validated scales were considerably more stable than complex models which were based on ad-hoc, single-study operationalizations. Taken together, these point to a single practical lesson. Replicability is perhaps not that mysterious. It is largely dependent on adequate power, realistic effect-size expectations, and proper measurement (as well as good theory, of course).

**Figure 5.** Statistical power of the original study against the proportion of paths that replicated, separately for exact replication and for same direction and significance (point size reflects the original sample size). The upward slope is the field’s reminder: underpowered originals replicate worst.

Exploring Analytical Robustness

By conducting several replications inside one survey, we had also a second, less common opportunity: we could swap measures and investigate what happens to the relationships. This is where the project stopped being a verification exercise and started saying something about our constructs. For example, when we exchanged one privacy-concern scale for another, or used perceived privacy risks instead of privacy concerns, the famous privacy paradox shifted dynamically. In our specification curve analysis of the concern-behavior link (see Figure 6), the single largest driver of whether concern predicted less or more disclosure was simply which of these two operationalizations we chose. This maps onto an earlier specification curve study on the same question (Masur, 2023), and here it holds even with validated multi-item scales rather than single survey items.

**Figure 6.** Results from the specification curve analysis in which measures were swapped between replicated studies to test the concern-disclosure relationship (Masur & Ranzini, 2025).

We should take this very seriously. A relationship whose sign flips depending on whether you measure “risk” or “concern” may not obviously describe one stable phenomenon. It may be reporting a measurement choice that the field has been treating as a theoretical puzzle. The privacy cynicism studies told a similar story: whether fatigue, resignation, or mistrust predicted protective behavior depended heavily on which control variables entered the model. Some of our most cited effects are robust. Others are conditional on decisions we rarely discuss in more depth.

There is something else to learn from these replications. When a finding fails to replicate, it is genuinely hard to tell whether the original was a false positive or whether the world has simply changed. Comparing data collected in 2016 with data collected in 2025, we saw real shifts in how people share. The between disclosing through private messages and disclosing through public status updates has largely vanished, plausibly because the platforms themselves changed, with Stories and other, more ephemeral formats redefining what “public” even means. So when a path does not replicate, the honest answer is often that we cannot say what the cause is. At times, it may be different measures, at other times, different populations. But more often than not, it may also be shifting cultures and norms, evolving platform architectures, or a new world. Only further replication and systematically exploring these causes, will tease them apart.

Conclusion

I want to highlight one benefit I did not anticipate when I started this project with students. Embedding replication in a MA course turned out to be an surprisingly good way to teach empirical science. It moves students out of the abstract textbook and drops them straight into the messy reality of our science. We had to grapple with undocumented procedures, measures that no longer fit the way they once did, or decisions the original authors never reported. Students who have wrestled with a real model that will not converge started to better understand power, measurement, and researcher degrees of freedom.

One more thing: I believe that no study is worth not to replicate. Even the non-contested, highly cited pillars of privacy research deserve a second look, and several of them did not replicate perfectly. The culture is shifting, slowly (Dienlin et al., 2020). Replication is no longer routinely dismissed as unoriginal, and a shared vocabulary for what counts as a replication is finally emerging (see also Vermeulen et al., 2024). But single-lab efforts like this one can only take us so far. To put online privacy science on a truly solid and replicable ground, we need to move toward large-scale, international, and many-labs collaborations.

So this is the modest case I wanted to make by doing rather than just proposing. Replication is feasible. In this project, we learned things about our constructs that no original study could have told us. Thus, let us fund, conduct, and fairly review replications!

References

Choi, H., Park, J., & Jung, Y. (2018). The role of privacy fatigue in online privacy behavior. Computers in Human Behavior, 81, 42–51.
Dienlin, T., Johannes, N., Bowman, N. D., Masur, P. K., Engesser, S., Kümpel, A. S., … de Vreese, C. (2021). An agenda for open science in Communication. Journal of Communication, 71(1), 1-26. https://doi.org/10.1093/joc/jqz052
Dienlin, T., & Trepte, S. (2015). Is the privacy paradox a relic of the past? An in-depth analysis of privacy attitudes and privacy behaviors. European Journal of Social Psychology, 45(3), 285–297.
Hallam, C., & Zanella, G. (2017). Online self-disclosure: The privacy paradox explained as a temporally discounted balance between concerns and rewards. Computers in Human Behavior, 68, 217–227.
Keating, D. M., & Totzkay, D. (2019). We do publish (conceptual) replications (sometimes): Publication trends in communication science, 2007–2016. Communication Studies, 70(1), 1–18.
Krasnova, H., Spiekermann, S., Koroleva, K., & Hildebrandt, T. (2010). Online social networks: Why we disclose. Journal of Information Technology, 25(2), 109–125.
LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018). A unified framework to quantify the credibility of scientific findings. Advances in Methods and Practices in Psychological Science, 1(3), 389–402.
Litt, E. (2013). Understanding social network site users’ privacy tool use. Computers in Human Behavior, 29(4), 1649–1656.
Lutz, C., Hoffmann, C. P., & Ranzini, G. (2020). Data capitalism and the user: An exploration of privacy cynicism in Germany. New Media & Society, 22(7), 1168–1187.
Masur, P. K. (2023). Understanding the effects of conceptual and analytical choices on “finding” the privacy paradox: A specification curve analysis of large-scale survey data. Information, Communication & Society, 26(8), 1631–1652.
Masur, P. K. (2026). Privacy fatigue and cynicism in the digital age: A close replication of Choi et al. (2018) and Lutz et al. (2020). In C. Lutz, C. P. Hoffmann, & A. Tamò-Larrieux (Eds.), Advancing the study of privacy cynicism, apathy and resignation in the digital society. Edward Elgar.
Masur, P. K., & Ranzini, G. (2025). Privacy calculus, privacy paradox, and context collapse: A replication of three key studies in communication privacy research. Journal of Communication. https://doi.org/10.1093/joc/jqaf007
Masur, P. K., & Scharkow, M. (2016). Disclosure management on social network sites: Individual privacy perceptions and user-directed privacy strategies. Social Media + Society, 2(1).
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Park, Y. J. (2013). Digital literacy and privacy behavior online. Communication Research, 40(2), 215–236.
Popper, K. R. (1959). The logic of scientific discovery. Hutchinson.
Vermeulen, I. E., Masur, P. K., Beukeboom, C. J., & Johnson, B. K. (2024). Direct replication in experimental communication science: A conceptual and practical exploration. Media and Communication. https://doi.org/10.17645/mac.7971
Vitak, J. (2012). The impact of context collapse and privacy on social network site disclosures. Journal of Broadcasting & Electronic Media, 56(4), 451–470.
Zlatolas, L. N., Welzer, T., Heričko, M., & Hölbl, M. (2015). Privacy antecedents for SNS self-disclosure: The case of Facebook. Computers in Human Behavior, 45, 158-167.