The big picture: Most findings survive

The overarching takeaway is quite reassuring, but perhaps also humbling. Cross-sectional, survey-based privacy findings broadly replicate. When we plot every original effect against its replication, the resulting scatterplot tracks the diagonal of perfect replication reasonably well, and most paths stay statistically significant (see Figure 1). Privacy research thus overall replicates. At least, if we look at the amount of individual paths that replicated in direction and significance. Yet, the devil lies in the details (read on for the surprises!).

One insights is that the effect sizes are slightly smaller on average and more wide-spread in the replications (see Figure 2). That shift should be familiar to anyone who has followed the reproducibility work in psychology (e.g., Open Science Collaboration, 2015). Even when an effect is real, the original literature tends to overestimate its size. An explanation might be that first significant results are the lucky ones, and luck does not (necessarily) replicate. So the honest summary is: the direction of our findings is mostly trustworthy, the exact effect sizes perhaps rather less so.

Not every finding is equally replicable

Averages hide the more interesting story, which is how much the studies differ from one another. At one end, some models mapped onto the new data almost flawlessly (e.g., Choi et al., 2018; Lutz et al., 2020; Hallam & Zanella, 2017). At the other, a few core assumptions turned out to be far more volatile (e.g., Dienlin & Trepte, 2015; Masur & Scharkow, 2016). When you separate “same direction and significance” from “exact replication” (the original estimate is inside the replication’s confidence interval, see Figure 3), the gap between the two bars is itself the finding: many studies kept most of their paths pointing the right way, while far fewer reproduced the original effect sizes.

That said, the overall good picture should not fool ourselves into believing that everything “replicates” (at least by direction and significance). At times, while the overall amount of paths that replicated was high, primary hypotheses (rather than paths e.g., related to control variables) did not replicate (e.g., Dienlin & Trepte, 2015; Zlatolas et al., 2017). Noteworthy, the often contested relationship between privacy concerns and self-disclosure (if non-significant mostly termed the “privacy paradox”) often did not replicate (e.g., Dienlin & Trepte, 2015; Vitak, 2012; Hallam & Zanella, 2017). Other times, key assumptions (e.g., benefits clearly predicting self-disclosure) also did not replicate (e.g., Krasnova et al., 2010). Looking at Figure 4, we do see quite some differences. Particularly if the scatterplots between original and replicated effect sizes widens, it deserves a closer look.

What makes a finding replicable? Predictors of replication success

Because we had ten studies rather than just one, we could ask questions that no single replication can answer: For example, what predicts whether a finding survives? Tentatively, three patterns stand out.

First, the statistical power of the original study seems to matter. The lower the original power, the less likely a path was to replicate, in direction and especially in magnitude (see Figure 5). Second, and relatedly, small original effects were the most likely to not replicate. Third, theoretical and measurement parsimony helped: studies built on a few clear mechanisms and previously validated scales were considerably more stable than complex models which were based on ad-hoc, single-study operationalizations. Taken together, these point to a single practical lesson. Replicability is perhaps not that mysterious. It is largely dependent on adequate power, realistic effect-size expectations, and proper measurement (as well as good theory, of course).

Exploring Analytical Robustness

By conducting several replications inside one survey, we had also a second, less common opportunity: we could swap measures and investigate what happens to the relationships. This is where the project stopped being a verification exercise and started saying something about our constructs. For example, when we exchanged one privacy-concern scale for another, or used perceived privacy risks instead of privacy concerns, the famous privacy paradox shifted dynamically. In our specification curve analysis of the concern-behavior link (see Figure 6), the single largest driver of whether concern predicted less or more disclosure was simply which of these two operationalizations we chose. This maps onto an earlier specification curve study on the same question (Masur, 2023), and here it holds even with validated multi-item scales rather than single survey items.

We should take this very seriously. A relationship whose sign flips depending on whether you measure “risk” or “concern” may not obviously describe one stable phenomenon. It may be reporting a measurement choice that the field has been treating as a theoretical puzzle. The privacy cynicism studies told a similar story: whether fatigue, resignation, or mistrust predicted protective behavior depended heavily on which control variables entered the model. Some of our most cited effects are robust. Others are conditional on decisions we rarely discuss in more depth.

There is something else to learn from these replications. When a finding fails to replicate, it is genuinely hard to tell whether the original was a false positive or whether the world has simply changed. Comparing data collected in 2016 with data collected in 2025, we saw real shifts in how people share. The between disclosing through private messages and disclosing through public status updates has largely vanished, plausibly because the platforms themselves changed, with Stories and other, more ephemeral formats redefining what “public” even means. So when a path does not replicate, the honest answer is often that we cannot say what the cause is. At times, it may be different measures, at other times, different populations. But more often than not, it may also be shifting cultures and norms, evolving platform architectures, or a new world. Only further replication and systematically exploring these causes, will tease them apart.

Conclusion

I want to highlight one benefit I did not anticipate when I started this project with students. Embedding replication in a MA course turned out to be an surprisingly good way to teach empirical science. It moves students out of the abstract textbook and drops them straight into the messy reality of our science. We had to grapple with undocumented procedures, measures that no longer fit the way they once did, or decisions the original authors never reported. Students who have wrestled with a real model that will not converge started to better understand power, measurement, and researcher degrees of freedom.

One more thing: I believe that no study is worth not to replicate. Even the non-contested, highly cited pillars of privacy research deserve a second look, and several of them did not replicate perfectly. The culture is shifting, slowly (Dienlin et al., 2020). Replication is no longer routinely dismissed as unoriginal, and a shared vocabulary for what counts as a replication is finally emerging (see also Vermeulen et al., 2024). But single-lab efforts like this one can only take us so far. To put online privacy science on a truly solid and replicable ground, we need to move toward large-scale, international, and many-labs collaborations.

So this is the modest case I wanted to make by doing rather than just proposing. Replication is feasible. In this project, we learned things about our constructs that no original study could have told us. Thus, let us fund, conduct, and fairly review replications!

References

• Choi, H., Park, J., & Jung, Y. (2018). The role of privacy fatigue in online privacy behavior. Computers in Human Behavior, 81, 42–51.
• Dienlin, T., Johannes, N., Bowman, N. D., Masur, P. K., Engesser, S., Kümpel, A. S., … de Vreese, C. (2021). An agenda for open science in Communication. Journal of Communication, 71(1), 1-26. https://doi.org/10.1093/joc/jqz052
• Dienlin, T., & Trepte, S. (2015). Is the privacy paradox a relic of the past? An in-depth analysis of privacy attitudes and privacy behaviors. European Journal of Social Psychology, 45(3), 285–297.
• Hallam, C., & Zanella, G. (2017). Online self-disclosure: The privacy paradox explained as a temporally discounted balance between concerns and rewards. Computers in Human Behavior, 68, 217–227.
• Keating, D. M., & Totzkay, D. (2019). We do publish (conceptual) replications (sometimes): Publication trends in communication science, 2007–2016. Communication Studies, 70(1), 1–18.
• Krasnova, H., Spiekermann, S., Koroleva, K., & Hildebrandt, T. (2010). Online social networks: Why we disclose. Journal of Information Technology, 25(2), 109–125.
• LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018). A unified framework to quantify the credibility of scientific findings. Advances in Methods and Practices in Psychological Science, 1(3), 389–402.
• Litt, E. (2013). Understanding social network site users’ privacy tool use. Computers in Human Behavior, 29(4), 1649–1656.
• Lutz, C., Hoffmann, C. P., & Ranzini, G. (2020). Data capitalism and the user: An exploration of privacy cynicism in Germany. New Media & Society, 22(7), 1168–1187.
• Masur, P. K. (2023). Understanding the effects of conceptual and analytical choices on “finding” the privacy paradox: A specification curve analysis of large-scale survey data. Information, Communication & Society, 26(8), 1631–1652.
• Masur, P. K. (2026). Privacy fatigue and cynicism in the digital age: A close replication of Choi et al. (2018) and Lutz et al. (2020). In C. Lutz, C. P. Hoffmann, & A. Tamò-Larrieux (Eds.), Advancing the study of privacy cynicism, apathy and resignation in the digital society. Edward Elgar.
• Masur, P. K., & Ranzini, G. (2025). Privacy calculus, privacy paradox, and context collapse: A replication of three key studies in communication privacy research. Journal of Communication. https://doi.org/10.1093/joc/jqaf007
• Masur, P. K., & Scharkow, M. (2016). Disclosure management on social network sites: Individual privacy perceptions and user-directed privacy strategies. Social Media + Society, 2(1).
• Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
• Park, Y. J. (2013). Digital literacy and privacy behavior online. Communication Research, 40(2), 215–236.
• Popper, K. R. (1959). The logic of scientific discovery. Hutchinson.
• Vermeulen, I. E., Masur, P. K., Beukeboom, C. J., & Johnson, B. K. (2024). Direct replication in experimental communication science: A conceptual and practical exploration. Media and Communication. https://doi.org/10.17645/mac.7971
• Vitak, J. (2012). The impact of context collapse and privacy on social network site disclosures. Journal of Broadcasting & Electronic Media, 56(4), 451–470.
• Zlatolas, L. N., Welzer, T., Heričko, M., & Hölbl, M. (2015). Privacy antecedents for SNS self-disclosure: The case of Facebook. Computers in Human Behavior, 45, 158-167.

Statistical background

Classical Test Theory (CTT) offers a framework where each observed score () is conceptualized as the sum of a true score () and the error score ():

In this context, the true score represents the actual level of the latent trait, and the error score encompasses all item-specific variance, measurement errors, or random noise.

CTT defines reliability as the proportion of variance in the observed scores that is attributable to the true scores. When items are randomly sampled, reliability can still be maintained by ensuring that the items are representative of the entire item pool and that the sampling process introduces minimal bias. The basic idea of item sampling is thus as follows:

• Random Selection of Items: Instead of administering the entire set of items (or a fixed subset) to every respondent, a small, randomly selected subset is given to each respondent, resulting in unique short scales per participant. This reduces the burden on respondents and shortens the questionnaire.
• Scoring with planned missing data: Because the items are missing completely at random (by design), we can estimate each person’s score from the items they answered, for example with a measurement model estimated by full information maximum likelihood (FIML). Alternatively, the unadministered items can be imputed. As we will see below, the first option is clearly preferable.

Despite the reduction in the number of items per respondent, the randomness and overlap between different respondents’ subsets ensure that the overall information about the latent trait is preserved at the level of the sample (e.g., means, variances, and correlations of the scale). For each individual person, however, the precision of their score depends on how many items they answered, as the following sections show.

# Set parameters
alpha <- .9
k_s <- 5
k <- 10

# Formula from above
new_alpha <- ((k_s/k)*alpha)/(1+((k_s/k)-1)*alpha)
round(new_alpha, 2)

#> [1] 0.82

library(tidyverse)

# Custom function to compute reliability and correlation
reli_new <- function(alpha, k_s, k, N = 100) {

  # Ratio between sample items and total items
  n <- k_s/k

  # Spearman-Brown formula
  alpha_new <- ((n)*alpha)/(1+((n)-1)*alpha)

  # Standard error of alpha_new (van Zyl, Neudecker & Nel, 2000)
  se_a <- (1 - alpha_new) * sqrt(2 * k_s / ((k_s - 1) * N))

  # Correlation between scales (part-whole correlation for parallel items)
  r <- sqrt(alpha_new/alpha)
  se_r <- (1-r^2)/sqrt(N)

  # Create output table
  tibble(N = N,
         k = k,
         k_s = k_s,
         n = n,
         alpha = alpha,
         alpha_new = alpha_new,
         se_a = se_a,
         r = r,
         se_r = se_r)
}

# Check function
(check <- reli_new(alpha = .9, k_s = 5, k = 10, N = 100))

#> # A tibble: 1 × 9
#>       N     k   k_s     n alpha alpha_new   se_a     r    se_r
#>   <dbl> <dbl> <dbl> <dbl> <dbl>     <dbl>  <dbl> <dbl>   <dbl>
#> 1   100    10     5   0.5   0.9     0.818 0.0287 0.953 0.00909

# Transparent background and muted text, so the figures work in light and dark mode
theme_blog <- function() {
  theme_minimal(base_size = 12) +
    theme(plot.background  = element_rect(fill = "transparent", colour = NA),
          panel.background = element_rect(fill = "transparent", colour = NA),
          panel.grid.minor = element_blank(),
          panel.grid.major = element_line(colour = "#8a8f9840", linewidth = .4),
          strip.text = element_text(colour = "#7a7f88", face = "bold"),
          axis.text  = element_text(colour = "#7a7f88"),
          axis.title = element_text(colour = "#7a7f88"),
          legend.text  = element_text(colour = "#7a7f88"),
          legend.title = element_text(colour = "#7a7f88"),
          legend.position = "bottom")
}
blues <- c("#c4e4f6", "#8cc8ec", "#4aa8e0", "#1680bd", "#0b5a87")   # sequential: alpha .75 -> .95

# Expected reliability for a 20-item pool across sampling ratios and original alphas
analytic <- expand.grid(alpha = seq(.75, .95, by = .05), k_s = 2:20, N = c(100, 1000)) |>
  pmap_dfr(reli_new, k = 20) |>
  mutate(N = factor(paste("N =", format(N, big.mark = ",")), levels = c("N = 100", "N = 1,000")))

ggplot(analytic, aes(n, alpha_new, colour = factor(alpha), fill = factor(alpha))) +
  geom_hline(yintercept = .8, linetype = "dashed", colour = "#8a8f98") +
  geom_ribbon(aes(ymin = alpha_new - 1.96 * se_a, ymax = alpha_new + 1.96 * se_a),
              alpha = .25, colour = NA) +
  geom_line(linewidth = 1) +
  facet_wrap(~ N) +
  scale_colour_manual(values = blues) +
  scale_fill_manual(values = blues) +
  scale_x_continuous(labels = scales::percent) +
  labs(x = "Share of items each person answers (k_s / k)",
       y = "Expected reliability (alpha)",
       colour = "Alpha of the full scale", fill = "Alpha of the full scale") +
  theme_blog()

ggplot(analytic, aes(n, r, colour = factor(alpha), fill = factor(alpha))) +
  geom_hline(yintercept = c(.8, .9), linetype = "dashed", colour = "#8a8f98") +
  geom_ribbon(aes(ymin = r - 1.96 * se_r, ymax = pmin(1, r + 1.96 * se_r)), alpha = .25, colour = NA) +
  geom_line(linewidth = 1) +
  facet_wrap(~ N) +
  scale_colour_manual(values = blues) +
  scale_fill_manual(values = blues) +
  scale_x_continuous(labels = scales::percent) +
  labs(x = "Share of items each person answers (k_s / k)",
       y = "Expected correlation with the full scale",
       colour = "Alpha of the full scale", fill = "Alpha of the full scale") +
  theme_blog()

Simulation approach

The key idea of the simulation is to generate data for which we know each person’s true score. Then we can measure directly how well the scores based on the sampled items recover it: the squared correlation between a score and the true score is exactly the reliability of that score. This is the same quantity the Spearman–Brown formula predicts, so we can compare the two.

library(lavaan)
library(mice)
library(furrr)

set.seed(123)  # for reproducibility

sim_items <- function(N, k, alpha, answer_options = 5) {

  # Average inter-item correlation implied by alpha, and the resulting loading
  rbar   <- alpha / (k - alpha * (k - 1))
  lambda <- sqrt(rbar)

  # True scores, item difficulties and continuous responses
  theta      <- rnorm(N)
  difficulty <- runif(k, -.75, .75)
  latent <- theta %o% rep(lambda, k) +
    matrix(rnorm(N * k, sd = sqrt(1 - lambda^2)), N, k) -
    matrix(difficulty, N, k, byrow = TRUE)

  # Cut into Likert-type answer options (1 to 5)
  cuts  <- qnorm(seq(0, 1, length.out = answer_options + 1))[2:answer_options]
  items <- matrix(findInterval(latent, cuts) + 1, N, k,
                  dimnames = list(NULL, paste0("i", 1:k)))

  list(items = as.data.frame(items), theta = theta)
}

# Check function
test <- sim_items(N = 100, k = 10, alpha = .9)
head(test$items)

#>   i1 i2 i3 i4 i5 i6 i7 i8 i9 i10
#> 1  3  1  2  1  2  1  3  3  2   5
#> 2  2  1  1  2  2  1  4  3  5   4
#> 3  4  3  5  5  2  2  4  5  5   5
#> 4  3  4  4  2  4  2  4  4  4   5
#> 5  5  2  5  3  3  1  4  4  4   4
#> 6  5  4  5  5  5  4  5  2  5   5

psych::alpha(test$items)$total

#>  raw_alpha std.alpha   G6(smc) average_r     S/N        ase  mean        sd
#>  0.8482603 0.8487234 0.8451815 0.3594018 5.61041 0.02258244 3.164 0.8487091
#>   median_r
#>  0.3625075

sample_items <- function(items, n) {
  k   <- ncol(items)
  k_s <- round(n * k)
  for (i in seq_len(nrow(items))) {
    items[i, sample(k, k - k_s)] <- NA
  }
  items
}

# Check function
test_na <- sample_items(test$items, n = .8)
head(test_na)

#>   i1 i2 i3 i4 i5 i6 i7 i8 i9 i10
#> 1  3  1  2  1 NA  1  3 NA  2   5
#> 2  2  1 NA  2  2  1 NA  3  5   4
#> 3  4  3  5  5  2 NA  4 NA  5   5
#> 4  3  4  4  2  4 NA  4  4  4  NA
#> 5  5  2  5  3  3  1  4 NA  4  NA
#> 6  5  4  5  5  5 NA  5  2  5  NA

mean(is.na(test_na))

#> [1] 0.2

score_mean <- function(items) rowMeans(items, na.rm = TRUE)

score_adjusted <- function(items) {
  centered <- sweep(as.matrix(items), 2, colMeans(items, na.rm = TRUE))
  rowMeans(centered, na.rm = TRUE)
}

score_cfa <- function(items) {
  model <- paste("G =~", paste(names(items), collapse = " + "))
  scores <- as.numeric(lavPredict(cfa(model, data = items, std.lv = TRUE)))
  # The sign of a factor is arbitrary: align it with the sum score
  # (all items are positively keyed)
  if (cor(scores, rowSums(items)) < 0) scores <- -scores
  scores
}

score_imputed <- function(items) {
  score_cfa(complete(mice(items, m = 1, printFlag = FALSE)))
}

# Reliability of each score = squared correlation with the true score
sapply(list(mean     = score_mean(test_na),
            adjusted = score_adjusted(test_na),
            imputed  = score_imputed(test_na)),
       \(score) cor(score, test$theta)^2) |> round(2)

#>     mean adjusted  imputed 
#>     0.79     0.80     0.76

alpha_full <- function(items) {
  k <- ncol(items)
  k / (k - 1) * (1 - sum(apply(items, 2, var)) / var(rowSums(items)))
}

sim_func <- function(N, k, alpha, n) {
  d <- sim_items(N, k, alpha)
  sampled <- sample_items(d$items, n)
  a <- alpha_full(d$items)

  full <- list(mean = score_mean(d$items), adjusted = score_mean(d$items),
               imputed = score_cfa(d$items))
  part <- list(mean = score_mean(sampled), adjusted = score_adjusted(sampled),
               imputed = score_imputed(sampled))

  tibble(N = N, k = k, k_s = round(n * k), n = n, alpha = alpha, alpha_emp = a,
         method = names(part),
         r_full = mapply(cor, part, full),
         reliability = sapply(part, cor, y = d$theta)^2,
         reliability_full = sapply(full, cor, y = d$theta)^2,
         spearman_brown = n * a / (1 + (n - 1) * a))
}

# Check sim function
sim_func(N = 500, k = 20, alpha = .85, n = .5)

#> # A tibble: 3 × 11
#>       N     k   k_s     n alpha alpha_emp method   r_full reliability
#>   <dbl> <dbl> <dbl> <dbl> <dbl>     <dbl> <chr>     <dbl>       <dbl>
#> 1   500    20    10   0.5  0.85     0.823 mean      0.912       0.684
#> 2   500    20    10   0.5  0.85     0.823 adjusted  0.926       0.707
#> 3   500    20    10   0.5  0.85     0.823 imputed   0.858       0.607
#> # ℹ 2 more variables: reliability_full <dbl>, spearman_brown <dbl>

# Setup simulation grid
setup <- expand.grid(
  N = c(250, 500, 1000),
  k = c(10, 20, 40),
  alpha = seq(.75, .95, by = .05),
  n = c(.3, .4, .5, .6, .7, .8)
)

# The simulation takes a while: its results are saved next to this post and
# re-used (delete sims/results.rds to run it again)
n_sim <- 200
if (file.exists("sims/results.rds")) {
  results <- readRDS("sims/results.rds")
} else {
  plan(multisession, workers = parallelly::availableCores(omit = 1))
  results <- setup[rep(seq_len(nrow(setup)), n_sim), ] |>
    future_pmap_dfr(sim_func, .options = furrr_options(seed = 2024))
  dir.create("sims", showWarnings = FALSE)
  saveRDS(results, "sims/results.rds")
}

results

#> # A tibble: 162,000 × 11
#>        N     k   k_s     n alpha alpha_emp method   r_full reliability
#>    <dbl> <dbl> <dbl> <dbl> <dbl>     <dbl> <chr>     <dbl>       <dbl>
#>  1   250    10     3   0.3  0.75     0.771 mean      0.780       0.408
#>  2   250    10     3   0.3  0.75     0.771 adjusted  0.811       0.451
#>  3   250    10     3   0.3  0.75     0.771 imputed   0.647       0.297
#>  4   500    10     3   0.3  0.75     0.718 mean      0.740       0.397
#>  5   500    10     3   0.3  0.75     0.718 adjusted  0.779       0.436
#>  6   500    10     3   0.3  0.75     0.718 imputed   0.508       0.165
#>  7  1000    10     3   0.3  0.75     0.727 mean      0.753       0.410
#>  8  1000    10     3   0.3  0.75     0.727 adjusted  0.781       0.435
#>  9  1000    10     3   0.3  0.75     0.727 imputed   0.597       0.251
#> 10   250    20     6   0.3  0.75     0.741 mean      0.777       0.438
#> # ℹ 161,990 more rows
#> # ℹ 2 more variables: reliability_full <dbl>, spearman_brown <dbl>

d <- filter(summ, N == 500, k == 20)

ggplot(d, aes(n, rel, colour = method)) +
  geom_line(aes(y = sb), colour = "#8a8f98", linetype = "dashed", linewidth = .8) +
  geom_pointrange(aes(ymin = rel_lo, ymax = rel_hi), size = .2,
                  position = position_dodge(width = .04)) +
  geom_line(position = position_dodge(width = .04)) +
  facet_wrap(~ lab, nrow = 1) +
  scale_colour_manual(values = method_cols, labels = method_labels) +
  scale_x_continuous(labels = scales::percent, breaks = c(.3, .5, .7)) +
  labs(x = "Share of items each person answers",
       y = "Reliability of the person scores",
       colour = NULL,
       caption = "Points: mean of 200 replications, bars: 95% of replications. Dashed: Spearman–Brown prediction.") +
  theme_blog() +
  theme(plot.caption = element_text(colour = "#7a7f88"))

ggplot(d, aes(n, r, colour = method)) +
  geom_line(aes(y = sqrt(sb / alpha_emp)), colour = "#8a8f98", linetype = "dashed", linewidth = .8) +
  geom_hline(yintercept = c(.8, .9), colour = "#8a8f9860") +
  geom_pointrange(aes(ymin = r_lo, ymax = r_hi), size = .2,
                  position = position_dodge(width = .04)) +
  geom_line(position = position_dodge(width = .04)) +
  facet_wrap(~ lab, nrow = 1) +
  scale_colour_manual(values = method_cols, labels = method_labels) +
  scale_x_continuous(labels = scales::percent, breaks = c(.3, .5, .7)) +
  labs(x = "Share of items each person answers",
       y = "Correlation with the full-scale score",
       colour = NULL) +
  theme_blog()

summ |>
  filter(method == "adjusted", alpha == .85, n == .5) |>
  ggplot(aes(factor(N), rel, colour = factor(k))) +
  geom_hline(aes(yintercept = mean(sb)), colour = "#8a8f98", linetype = "dashed") +
  geom_pointrange(aes(ymin = rel_lo, ymax = rel_hi), position = position_dodge(width = .5)) +
  scale_colour_manual(values = c("10" = "#8cc8ec", "20" = "#1497d6", "40" = "#0b5a87")) +
  labs(x = "Sample size", y = "Reliability", colour = "Items in the pool") +
  theme_blog()

summ |>
  filter(N == 500, k == 20, n == .6, alpha == .8) |>
  mutate(method = method_labels[as.character(method)]) |>
  select(method, alpha_emp, `Spearman-Brown` = sb, reliability = rel, r_full = r)

#> # A tibble: 3 × 5
#>   method                           alpha_emp `Spearman-Brown` reliability r_full
#>   <chr>                                <dbl>            <dbl>       <dbl>  <dbl>
#> 1 Difficulty-adjusted mean             0.768            0.665       0.663  0.931
#> 2 Imputation (PMM) + factor scores     0.768            0.665       0.555  0.855
#> 3 Mean of answered items               0.768            0.665       0.646  0.918

# Mathematical results (with the alpha of the simulated Likert-type data)
a <- unique(round(filter(summ, N == 500, k == 20, n == .6, alpha == .8)$alpha_emp, 3))
reli_new(alpha = a, k_s = 12, k = 20, N = 500) |>
  mutate(ll = r - 1.96*se_r,
         ul = r + 1.96*se_r) |>
  dplyr::select(N, k, k_s, alpha, alpha_new, r, ll, ul)

#> # A tibble: 1 × 8
#>       N     k   k_s alpha alpha_new     r    ll    ul
#>   <dbl> <dbl> <dbl> <dbl>     <dbl> <dbl> <dbl> <dbl>
#> 1   500    20    12 0.768     0.665 0.931 0.919 0.942

Conclusion and recommendations

So what does this all mean? From my point of view, the simulations indicate that item sampling is a viable strategy for efficiently measuring latent traits in social science research (e.g., scales of 10-40 items with Likert-type items):

• The Spearman–Brown formula is a good planning tool. Given the alpha of the full scale and the share of items each person answers, it predicts the reliability of the person scores well. Just make sure to plug in the alpha of your actual (Likert-type) data. If the full scale has an alpha of about .90 and people answer half of the items, their scores still have a reliability of about 0.78 and correlate r = 0.94 with the full-scale scores. With a full-scale alpha of .80 to .85, half of the items yields reliabilities of only about 0.62 to 0.70; to reach .80, people then need to answer most of the items.
• Score people on the items they answered; don’t impute. A (difficulty-adjusted) mean or a model-based score from the answered items (e.g., FIML or IRT) is clearly better than imputing the missing items first. Imputation makes the data set look complete, but it adds noise to every person’s score.
• Sample size is about precision, not about reliability. Larger samples estimate everything more precisely, but they do not make each person’s score more reliable.

This technique reduces respondent burden and shortens questionnaires. At the level of the sample (e.g., means and correlations of the scale), item sampling preserves the information of the full item pool. At the level of individuals, the reliability of their scores follows Spearman–Brown, and that is something we can plan for.

References

• de Vet, H. C., Mokkink, L. B., Mosmuller, D. G., & Terwee, C. B. (2017). Spearman–Brown prophecy formula and Cronbach’s alpha: different faces of reliability and opportunities for new applications. Journal of Clinical Epidemiology, 85, 45-49.
• Husek, T. R, & Sirotnik, K. (1967). Item Sampling in Educational Research. https://files.eric.ed.gov/fulltext/ED013975.pdf
• van Zyl, J. M., Neudecker, H., & Nel, D. G. (2000). On the distribution of the maximum likelihood estimator of Cronbach’s alpha. Psychometrika, 65(3), 271-280.

sessionInfo()

#> R version 4.4.0 (2024-04-24)
#> Platform: aarch64-apple-darwin20
#> Running under: macOS 26.6.2
#> 
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib 
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
#> 
#> locale:
#> [1] C.UTF-8/C.UTF-8/C.UTF-8/C/C.UTF-8/C.UTF-8
#> 
#> time zone: Europe/Amsterdam
#> tzcode source: internal
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#>  [1] furrr_0.3.1     future_1.70.0   mice_3.16.0     lavaan_0.6-19  
#>  [5] lubridate_1.9.3 forcats_1.0.0   stringr_1.6.0   dplyr_1.2.0    
#>  [9] purrr_1.2.1     readr_2.1.5     tidyr_1.3.2     tibble_3.3.1   
#> [13] ggplot2_4.0.2   tidyverse_2.0.0
#> 
#> loaded via a namespace (and not attached):
#>  [1] gtable_0.3.6       shape_1.4.6.1      xfun_0.57          htmlwidgets_1.6.4 
#>  [5] psych_2.4.3        lattice_0.22-6     tzdb_0.4.0         quadprog_1.5-8    
#>  [9] vctrs_0.7.2        tools_4.4.0        Rdpack_2.6.6       generics_0.1.4    
#> [13] parallel_4.4.0     stats4_4.4.0       pan_1.9            pkgconfig_2.0.3   
#> [17] jomo_2.7-6         Matrix_1.7-0       RColorBrewer_1.1-3 S7_0.2.1          
#> [21] lifecycle_1.0.5    compiler_4.4.0     farver_2.1.2       mnormt_2.1.1      
#> [25] codetools_0.2-20   htmltools_0.5.8.1  yaml_2.3.12        glmnet_4.1-8      
#> [29] nloptr_2.2.1       pillar_1.11.1      MASS_7.3-60.2      reformulas_0.4.4  
#> [33] iterators_1.0.14   rpart_4.1.23       boot_1.3-30        foreach_1.5.2     
#> [37] mitml_0.4-5        parallelly_1.46.1  nlme_3.1-164       tidyselect_1.2.1  
#> [41] digest_0.6.39      stringi_1.8.7      listenv_0.10.1     labeling_0.4.3    
#> [45] splines_4.4.0      fastmap_1.2.0      grid_4.4.0         cli_3.6.5         
#> [49] magrittr_2.0.4     utf8_1.2.6         survival_3.5-8     broom_1.0.12      
#> [53] pbivnorm_0.6.0     withr_3.0.2        scales_1.4.0       backports_1.5.0   
#> [57] timechange_0.3.0   rmarkdown_2.30     globals_0.19.1     nnet_7.3-19       
#> [61] lme4_2.0-1         hms_1.1.3          evaluate_1.0.5     knitr_1.51        
#> [65] rbibutils_2.4.1    rlang_1.1.7        Rcpp_1.1.1         glue_1.8.0        
#> [69] minqa_1.2.8        jsonlite_2.0.0     R6_2.6.1

A simple example (using simulations)

Power can best be understood through an example. Let’s imagine that we are interested in testing whether people perceive a stronger norm to disclose themselves on social media, when the majority of other users disclose themselves (cf. the study by Masur, DiFranzo & Bazarova, 2021). In this experiment, participants are exposed to a social media feed that varies with regard to how many of the users disclosed themselves visually (5% vs. 80%). Afterwards, we measure norm perceptions. The assumption is that the more people disclose themselves, the higher the perceived norm.

Just because we can, we will simulate the true effect in the population from which the draw our sample in the experiment:

# Load package
library(tidyverse)

# Set seed to ensure reproducibility
set.seed(5)

# Simulate true effect in the population
n <- 1e5
control <- rnorm(n, 4, 1)
treatment <- rnorm(n, 5, 1)
pop <- tibble(condition = rep(c("control", "treatment"), each = n),
              outcome = c(control, treatment))
head(pop)

#> # A tibble: 6 × 2
#>   condition outcome
#>   <chr>       <dbl>
#> 1 control      3.16
#> 2 control      5.38
#> 3 control      2.74
#> 4 control      4.07
#> 5 control      5.71
#> 6 control      3.40

This data set now represent the population from which we draw and how they would react to the different stimuli. Let’s quickly plot the distributions to get a better feel for the strength effect.

p1 <- ggplot(pop, aes(x = outcome, fill = condition)) +
  geom_density(alpha = .3) +
  theme_void() +
  theme(legend.position = "bottom")
p1

We can see that two distributions overlap considerably, but the central tendencies clearly suggest a difference. Now, let’s conduct our experiment. We sample n = 10 people from this population, which are by
definition randomly either in the control or in the treatment condition.

set.seed(11)
exp <- sample_n(pop, size = 10)
head(exp)

#> # A tibble: 6 × 2
#>   condition outcome
#>   <chr>       <dbl>
#> 1 control      4.10
#> 2 treatment    4.76
#> 3 control      3.82
#> 4 control      6.62
#> 5 control      3.09
#> 6 treatment    2.92

Let’s see how these 10 people map onto the true population. We can do so by plotting them as points on the x-axis.

p1 +
  geom_point(aes(x = outcome, y = 0, fill = condition),
             data = exp, alpha = .8, shape = 21, size = 5, color = "black")

It seems we got a bit unlucky. The difference that is clearly visible in the population is not as apparent in the sample. Let’s confirm our suspicion and actually test the difference using a standard t-test:

t.test(outcome ~ condition, exp)

#> 
#>  Welch Two Sample t-test
#> 
#> data:  outcome by condition
#> t = -0.81441, df = 5.688, p-value = 0.4482
#> alternative hypothesis: true difference in means between group control and group treatment is not equal to 0
#> 95 percent confidence interval:
#>  -3.225805  1.630822
#> sample estimates:
#>   mean in group control mean in group treatment 
#>                3.994465                4.791956

Indeed, the difference is not significant, even though we know that the effect does exist in the population. So what went wrong? Well, the study was underpowered. In other words, the sample size was too small to have a sufficiently high change to detect the effect.

So what would happen if we sample more people (e.g., n = 100)?

set.seed(11)
exp2 <- sample_n(pop, size = 100)

p1 +
  geom_point(aes(x = outcome, y = 0, fill = condition),
             data = exp2, alpha = .8, shape = 21, size = 5, color = "black")

Now, it seems that the two groups (shown as circles in the figure above) are more “in line” with the distribution of the population. Let’s again test the mean difference and voila, the difference now is significant.

t.test(outcome ~ condition, exp2)

#> 
#>  Welch Two Sample t-test
#> 
#> data:  outcome by condition
#> t = -4.67, df = 91.213, p-value = 1.033e-05
#> alternative hypothesis: true difference in means between group control and group treatment is not equal to 0
#> 95 percent confidence interval:
#>  -1.3850141 -0.5584102
#> sample estimates:
#>   mean in group control mean in group treatment 
#>                4.047700                5.019412

What we can learn from this example is the following: The probability of finding a significant effect based on our sample (given that the effect actually exist) is contingent on several aspects:

• the size of our sample (n1 =10, n2 = 100)
• the strength of the true effect
• the significance level (if we would have chosen a stricter level, such as e.g., 1%, the probability of finding a significant effect would be lower)
• the type of analysis the we are conducting (in this case a t-test)

And the probability of finding a significant effect given these parameters is what we are going to talk about in this chapter: it is what we call statistical power!

What is power?

The concept “statistical power” refers to the probability of correctly rejecting a null hypothesis (H0) when the
alternative hypothesis (H1) is true. It can be formally written as 1-beta, and represents the chances of a true positive detection conditional on the actual existence of an effect to detect. As the power of a test increases, the probability of making a type II error, which means wrongly failing to reject the null hypothesis decreases (false negative result). In simpler words, the more power we have, the less likely we are to assume that there is no effect, even though the effect actually exists (see Figure 1).

Figure 1: Difference between type I and type II error

Power is determined by several factors that are often contingent on the exact study that is conducted. But in any case, it is primarily determined by three factors:

—

1. The effect size
2. The significance level
3. The sample size

—

The effect size can be a direct value of the quantity of interest (e.g, b) in a regression analysis or the mean difference in a t-test), but it can also be a standardized measure that also accounts for the variability in the population (e.g., beta) in regression analysis or Cohen’s d in a t-test). In a power analysis (see further below), this parameter is oft utmost interest. Before we collect data, we can only guess the magnitude of the effect. At times, we might have information from prior research that informs our estimation. But even if no similar study exists, we can make informed guesses. For example, Rain, Levine & Weber (2018) found an average effect size of r = .21 across 149 meta-analyses spanning 60 years of communication research. This could be a useful baseline for a power analysis (if one can reasonably assume that one’s own study is not different than the average communication science study!). Another alternative would be to consider the smallest effect size that one would still consider sufficient support for the hypothesis under investigation. For example, Purington et al. (2021) investigate the impact of a experiential learning platform on subsequent knowledge increase as assessed in a 8-item knowledge test. They argued that the smallest (unstandardized) effect size that would still speak for the effectiveness of the platform corresponds to participants who received the intervention having 1 more correct answer in the test compared to this who did not receive the intervention. The concept of the smallest effect size of interest (SESOI) can also be applied more generally. For a given hypothesis, what would be the smallest standardized effect size that still supports the hypothesis? Often, we may then resort to an effect size of e.g., r = |.10|, which represents a “small” effect according to Cohen (1988).

The significance level refers to the false positive rate and is usually set to 5%. One way to increase the power of a hypothesis test would be to use a larger significance level (e.g., 10%). This increases the chance of rejecting the null hypothesis (obtaining a statistically significant result) when the null hypothesis is false; that is, the effect actually exists. But, at the same time, it heightens the risk of obtaining a statistically significant result when the null
hypothesis is not false; that is, the effect does not actually exist. Thus, although a less conservative significance level may increase power to detect a statistically significant result when the effect is true (lower false negative rate), it heightens the probability of obtaining a statistically significant effect when the effect does not exist (higher false positive rate).

The sample size greatly determines power as it affects the sampling error inherent in a test. The larger the sample size, the more it resembles the population from which it was drawn. If all other factors are kept equal, effects are generally harder to detect in smaller samples. Thus increasing the sample size is often the easiest way to boost the statistical power of a test.

Try it yourself: change the true effect, the sample size and the significance level, then run a few studies and see how often they come out significant.

If you prefer code: the cell below runs in your browser (via webR), so you can edit it and press Run Code. It simulates 1,000 studies and compares the share of significant results with the analytical answer from power.t.test().

Given the entanglement of these factors with power, we can compute each of them based on all others. So if the effect size, the sample size, and the significance level is known, we can compute the power. And
vice versa, if we know the effect size, the significance level and the power, we can compute the necessary sample size. This is used in the so-called “power analysis”, which we discuss in the next section.

Before we do so, however, we have to note that next to these three pivotal factors, there are also other aspects influencing the power of a hypothesis test. For example, measurement precision also affects power. The better or measures (i.e., higher reliability), the higher the power. Another influential factor is the design of the study and what analysis is used (see further below).

Power analysis

A power analysis most often refers to calculating the minimum sample size required to detect an effect of a certain magnitude given a certain significance level and power goal. For example, before I collect my data, I would like to know how many participants I should recruit. Given the norm of the field, I set the significance level to 5%. I further have heard that I should test my effect with an equally strict power, so I set the power to 95%. Based on prior research, I assume that the effect size of interest should be r = .10. The type of analysis is a correlation analysis. So, I know all three important factors as well as the type of analysis. How can I compute the necessary sample size? The formula for calculating the necessary sample size for a correlation test is:

where n is the sample size, Z~1-alpha/2~ is the z-score corresponding to the desired level of significance, Z1-beta is the z-score corresponding to the desired power of the test, and r is the expected magnitude of the correlation between the two variables. For our example, it can thus be computed like so:

This means that the study will need to have a sample size of at least 1,296 in order to have a power of 95%, a significance level of 5%, and an effect size of r = 0.10. As mentioned earlier, this is just an approximation, and the actual sample size needed may vary depending on the specific characteristics of the study.

However, it has to be noted that the example above is what we would call an “a priori power analysis”. The term “power analysis” is broader and refers to at least three types of analyses that differ with regard to when they are usually conducted and their respective goals (see Table below).

Power analysis in R

There are different ways to compute power (or rather the necessay sample size) in R. Very broadly speaking, we can differentiate between calculating power (e.g., with packages such as pwr) or by simulating power (e.g., with packages such as MonteCarlo. In the following, I will exemplify both approaches.

# Load package
library(pwr)

# Computing Cohen's d based on the results of our first experiment
cohens_d <- (4.79-3.99)/1.43

# Estimate power
pwr.t.test(d = cohens_d,                # Effect size
           n = 5,                       # Sample size per group
           sig.level = .05,             # Alpha error
           alternative = "two.sided")   # Type of test

#> 
#>      Two-sample t test power calculation 
#> 
#>               n = 5
#>               d = 0.5594406
#>       sig.level = 0.05
#>           power = 0.12256
#>     alternative = two.sided
#> 
#> NOTE: n is number in *each* group

# Computing Cohen's d based on the results of our first experiment
cohens_d <- (5.02-4.05)/1.13

# Post-hoc power analysis
pwr.t.test(d = cohens_d,
           n = 50,
           sig.level = .05,
           alternative = "two.sided")

#> 
#>      Two-sample t test power calculation 
#> 
#>               n = 50
#>               d = 0.8584071
#>       sig.level = 0.05
#>           power = 0.9889793
#>     alternative = two.sided
#> 
#> NOTE: n is number in *each* group

# Estimate necessary sample size
pwr.r.test(r = .1,                      # Assumed effect size
           sig.level = .05,             # alpha error
           power = .95,                 # Power (1 - beta error)
           alternative = "two.sided")   # Type of test

#> 
#>      approximate correlation power calculation (arctangh transformation) 
#> 
#>               n = 1292.88
#>               r = 0.1
#>       sig.level = 0.05
#>           power = 0.95
#>     alternative = two.sided

# Set power range
power <- seq(.50, .95, by = .05)

# Create function to extract sample size from pwr.r.test
extract_n <- function(x) {
  pwr.res <- pwr.r.test(r = .1,                      
                        sig.level = .05,            
                        power = x,                 
                        alternative = "two.sided")
  pwr.res$n
}

# Combine in table
pwr.table <- tibble(power = power,
       sample_size = power |> 
  map_dbl(function(x) extract_n(x)))
pwr.table

#> # A tibble: 10 × 2
#>    power sample_size
#>    <dbl>       <dbl>
#>  1  0.5         384.
#>  2  0.55        434.
#>  3  0.6         489.
#>  4  0.65        548.
#>  5  0.7         615.
#>  6  0.75        692.
#>  7  0.8         782.
#>  8  0.85        894.
#>  9  0.9        1046.
#> 10  0.95       1293.

# Plot sample curve
pwr.table |> 
  ggplot(aes(x = sample_size, y = power)) +
  geom_line(color = "red") +
  geom_hline(yintercept = .8, linetype = "dashed", color = "grey") +
  theme_classic() +
  labs(x = "Sample Size", y = "Power")

power <- seq(.60, .95, by = .05)
effect <- seq(.1, .5, by = .05)
grid <- expand_grid(power, effect)

# Extract sample sizes
extract_param <- function(x, y) {
  pwr.res <- pwr.r.test(r = y,                      
                        sig.level = .05,            
                        power = x,                 
                        alternative = "two.sided")
  pwr.res$n
}

# Plot curves
tibble(grid, 
       n = pmap_dbl(list(grid$power, grid$effect), extract_param)) |> 
  ggplot(aes(x = n, y = power, group = effect, color = effect)) +
  geom_hline(yintercept = .8, linetype = "dashed", color = "grey") +
  geom_line() +
  theme_classic() +
  labs(x = "Sample Size", y = "Power")

# Load package
library(MonteCarlo)

# Ensure reproducibility
set.seed(42)

# Define parameters
n <- 500
means <- c(2.5, 2.75, 3, 4)  # a1b1, a2b1, a1b2, a2b2
sd <- 1

# Simulate data
dv <- rnorm(n, mean = means , sd = sd) %>% round(0)
iv1 <- rep(c("a1", "a2"),each = 1, n/2)
iv2 <- rep(c("b1", "b2"), each = 2, n/4)
d <-data.frame(dv, iv1, iv2)

# Check data simulation
d %>%
  group_by(iv1, iv2) %>%
  summarise(m = mean(dv)) %>%
  ggplot(aes(x = factor(iv1), y = m, color = iv2, group = iv2)) +
  geom_point(size = 4) +
  geom_line(linewidth = 1) +
  ylim(1, 5) +
  theme_classic() +
  labs(y = "DV", x = "IV 1",
       color = "IV 2")

# Simulation function
sim_func <- function(n = 500, 
                     means = c(2.5, 2.75, 3, 4),
                     sd = 1) {
  
  # Simulate data
  dv <- round(rnorm(n, mean = means , sd = sd),0)
  iv1 <- rep(c("a1", "a2"),each = 1, n/2)
  iv2 <- rep(c("b1", "b2"), each = 2, n/4)
  d <-data.frame(dv, iv1, iv2)
  
  # Fit models
  fit1 <- lm(dv ~ iv1, d)     # Main effet of IV1
  fit2 <- lm(dv ~ iv2, d)     # Main effect of IV2
  fit3 <- lm(dv ~ iv1*iv2, d) # Interaction between both
  
  # Extract p-values and compute significance
  p_1 <- summary(fit1)$coef[2,4]
  sig_1 <- ifelse(p_1 < .05, TRUE, FALSE)
  p_2 <- summary(fit2)$coef[2,4]
  sig_2 <- ifelse(p_2 < .05, TRUE, FALSE)
  p_3 <- summary(fit3)$coef[4,4]
  sig_3 <- ifelse(p_3 < .05, TRUE, FALSE)
  

  # return values as list
  return(list("p_1" = p_1,
              "sig_1" = sig_1,
              "p_2" = p_2,
              "sig_2" = sig_2,
              "p_3" = p_3,
              "sig_3" = sig_3))
}

# Parameters that should vary across simulations
n_grid <- seq(100, 600, 100)
sd_grid <- c(1, 1.5, 2)

# Collect simulation parameters in list
(param_list <- list("n" = n_grid,
                    "sd" = sd_grid))

#> $n
#> [1] 100 200 300 400 500 600
#> 
#> $sd
#> [1] 1.0 1.5 2.0

result <- MonteCarlo(func = sim_func,            # pass test function
                     nrep = 1000,                 # number of tests
                     ncpus = 4,                   # number of cores to be used
                     param_list = param_list, )   # provide parameters

df <- MakeFrame(result)
head(df)

#>     n sd          p_1 sig_1          p_2 sig_2          p_3 sig_3
#> 1 100  1 2.482367e-03     1 3.051370e-05     1 1.254294e-01     0
#> 2 200  1 5.234942e-03     1 5.559082e-08     1 5.141067e-04     1
#> 3 300  1 5.229393e-07     1 2.278427e-13     1 1.114671e-01     0
#> 4 400  1 3.545625e-08     1 3.673009e-11     1 1.094738e-05     1
#> 5 500  1 1.240856e-06     1 2.743887e-20     1 6.631778e-05     1
#> 6 600  1 1.895437e-12     1 1.446259e-19     1 1.019437e-03     1

df %>%
  dplyr::select(-contains("p")) %>%
  gather(key, value, -n, -sd) %>%
  group_by(n, sd, key) %>%
  summarize(power = sum(value)/length(value)*100) %>%
  ggplot(aes(x = n, y = power, color = key)) +
  geom_line() +
  geom_point() +
  geom_hline(yintercept = 80, linetype = "dashed") +
  geom_hline(yintercept = 95, linetype = "dashed") +
  facet_wrap(~sd) +
  theme_bw() +
  labs(title = "Power Curves",
       x = "sample size (n)", 
       y = "power (1-beta)", 
       color = "type of effect",
       caption = "Note: facets represent different standard deviations")

Introduction

IRT refers to a set of mathematical models which aim to explain the relationship between a latent ability, trait, or proficiency (denoted ) and its observable manifestations (e.g., multiple-choice questions, true-false items, items…). In contrast to classical test theory (CTT), IRT focuses on the pattern of responses and considers responses in probabilistic terms, rather than focusing on composite variables and linear regression theory. IRT thereby accounts for…

• item discrimination: ability of an item to differentiate between respondents with different levels of proficiency.
• item difficulty: the likelihood of a correct response, expressed as the proficiency level at which 50% of the participant sample is estimated to answer an item correctly.
• Depending on the model some other parameters such as e.g., guessing probability…

Assessing item difficulties is useful in matching the test and trait levels of a target population and in ensuring that the entire range of the proficiency is covered. Therefore, there are several advantages to IRT models (compared to CTT models):

1. The stronger focus on item difficulty generally leads to scales that perform better at differentiating at the extremes of proficiency.

2. Item banks that are developed using IRT often provide a good basis for creating smaller scales or parallel tests that still exhibit a high validity and reliability.

3. It allows for computer adaptive testing (CAT). This approach can greatly approach the efficiency of testing. In CAT, the test items are selected to match each individual’s proficiency, so that the s/he will not be bored by easy items or frustrated by overly difficult items.

4. Specific IRT models (e.g., the Rasch model, see further below) has specific mathematical properties that are highly desirable, such as a) the number-correct score is a sufficient estimation of , b) specific objectivity, which means that item and person parameters will be similar even if only a subset of the item pool is used or a different population is studied.

The theoretical background of IRT models is to comprehensive to be covered here. Rather, this tutorial aims to introduce the main models and discuss their properties. For further study, we refer to the excellent book “Item Response Theory” by Christine DeMars. In this short introductory tutorial, we focus on three of the most used IRT models: the 3PL model, the 2PL model, and finally the 1PL or Rasch model. These models are named by the number of item parameter used in the function that models the relationship between and the item response (0/1). Each model has unique properties but all of them are suited to estimate latent variables from binary items (e.g., knowledge tests), which we will deal with in this tutorial. There are also more complex IRT models, such as e.g., graded response models, which can be used for non-binary items (e.g., likert-type scales). Yet, these will be discussed in a more advanced tutorial.

Preparation and Data

Although we are going to focus on the the package mirt in this tutorial, there are actually several packages that can be used to estimated IRT models. The most common ones are ltm, eRm, or TAM. This article by Choi & Asilkalkan (2019) also provides an overview of all available packages and their specific advantages: https://doi-org.vu-nl.idm.oclc.org/10.1080/15366367.2019.1586404. For this tutorial, we are going to load the tidyverse (for data wrangling and visualization) and mirt (for the main IRT analyses). And as noted before, we further are going to load my package ggmirt, which represents an extension to mirt and provides functions to plot more publication-ready figures and helps to assess item, person, and model fit.

# Data wrangling
library(tidyverse)

# Very comprehensive package for IRT analyses
library(mirt)

# Extension for 'mirt'
# devtools::install_github("masurp/ggmirt")
library(ggmirt)

For ggmirt package includes a convenient function to simulate data for IRT analyses. Let’s quickly create a data set with 500 observations and 10 items that should fit a 3PL, 2PL and perhaps even a 1PL model.

set.seed(42)
d <- sim_irt(500, 10, discrimination = .25, seed = 42)
head(d)

As you can see, each “participants” has answered 10 items that are binary. Imagine we would have administered a test (e.g., LSAT, PISA, knowledge test, exam,…) to 500 people. The score 1 means this person has answered a particular question/item correctly. The score 0 means, this person has answered this item falsely.

3PL model

The 3PL model takes item discrimination (first parameter: a), item difficulty (second parameter: b), and guessing probability (third parameter: c) into account. As such, the 2PL and 1PL model (discussed below, are special cases, or constrained versions of the 3PL model). Take a look at Fig. 1 below. It shows a typical item characteristic curve (ICC, but not to be mistaken for the intra-class correlation). The x-axis shows the latent ability () ranging from -4 to 4, with 0 being the average ability in the studied population. The y-axis shows the probability of solving the item. The curve thus represents the probability of answering this item given a certain level on the latent ability.

In this example, this 3PL model provides a smooth curve for this item because it is based on item discrimination (steepness of the slope), difficulty (point of inflexion at which the probability of answering the item correctly is 50%) and the guessing probability (slightly raised lower asymptote). The mathematical form of the 3PL model is:

Here, the probability of a correct response given a person’s ability is expressed as a function of all three item parameters and a person’s ability, where indicates the item and indicates an individual person. In the figure, the steeper the slope, the better the item is at differentiating people clearly in close proximity to its difficulty. The lower the asymptote, the lower the likelihood of selecting the right response by chance. As the item difficulty (point of inflexion is slightly above the average: 0.71), the item can be solved by a majority of potential participants.

So how do we fit such a model?

unimodel <- 'F1 = 1-10'

fit3PL <- mirt(data = d,
               model = unimodel,  # alternatively, we could also just specify model = 1 in this case
               itemtype = "3PL",
               verbose = FALSE)
fit3PL

#> 
#> Call:
#> mirt(data = d, model = unimodel, itemtype = "3PL", verbose = FALSE)
#> 
#> Full-information item factor analysis with 1 factor(s).
#> Converged within 1e-04 tolerance after 24 EM iterations.
#> mirt version: 1.45.1 
#> M-step optimizer: BFGS 
#> EM acceleration: Ramsay 
#> Number of rectangular quadrature: 61
#> Latent density type: Gaussian 
#> 
#> Log-likelihood = -2738.689
#> Estimated parameters: 30 
#> AIC = 5537.377
#> BIC = 5663.816; SABIC = 5568.594
#> G2 (993) = 497.87, p = 1
#> RMSEA = 0, CFI = NaN, TLI = NaN

# Factor solution
summary(fit3PL)

#>        F1    h2
#> V1  0.707 0.500
#> V2  0.513 0.264
#> V3  0.535 0.286
#> V4  0.727 0.529
#> V5  0.574 0.329
#> V6  0.566 0.320
#> V7  0.603 0.364
#> V8  0.443 0.196
#> V9  0.718 0.516
#> V10 0.480 0.230
#> 
#> SS loadings:  3.534 
#> Proportion Var:  0.353 
#> 
#> Factor correlations: 
#> 
#>    F1
#> F1  1

params3PL <- coef(fit3PL, IRTpars = TRUE, simplify = TRUE)
round(params3PL$items, 2) # g = c = guessing parameter

#>        a     b    g u
#> V1  1.70  1.17 0.00 1
#> V2  1.02 -0.58 0.00 1
#> V3  1.08  0.35 0.00 1
#> V4  1.80  0.81 0.09 1
#> V5  1.19  0.52 0.00 1
#> V6  1.17 -0.14 0.00 1
#> V7  1.29  1.87 0.00 1
#> V8  0.84  0.03 0.00 1
#> V9  1.76  2.11 0.00 1
#> V10 0.93 -0.05 0.01 1

M2(fit3PL)

itemfit(fit3PL)

itemfit(fit3PL, fit_stats = "infit") # typical for Rasch modeling

itemfitPlot(fit3PL)

head(personfit(fit3PL))

personfit(fit3PL) %>%
  reframe(infit.outside = prop.table(table(z.infit > 1.96 | z.infit < -1.96)),
          outfit.outside = prop.table(table(z.outfit > 1.96 | z.outfit < -1.96))) # lower row = non-fitting people

personfitPlot(fit3PL)

itempersonMap(fit3PL)

tracePlot(fit3PL)

tracePlot(fit3PL, facet = F, legend = T) + scale_color_brewer(palette = "Set3")

# Plotting only individual items
tracePlot(fit3PL,items = c(1:3), facet = F, legend = T) + scale_color_brewer(palette = "Set2")

itemInfoPlot(fit3PL) + scale_color_brewer(palette = "Set3")

itemInfoPlot(fit3PL, facet = T)

testInfoPlot(fit3PL, adj_factor = 2)

scaleCharPlot(fit3PL)

score <- fscores(fit3PL)
sumscore <- rowSums(d)
cor.test(score, sumscore)

#> 
#>  Pearson's product-moment correlation
#> 
#> data:  score and sumscore
#> t = 181.54, df = 498, p-value < 2.2e-16
#> alternative hypothesis: true correlation is not equal to 0
#> 95 percent confidence interval:
#>  0.9910997 0.9937302
#> sample estimates:
#>       cor 
#> 0.9925295

2PL model

The 2PL model only differs from the 3PL model in one regard: All items are assumed to have no guessing probability. So the model only takes item discrimination (a) and item difficulty into account. The mathematical form is hence (we simply delete the part before the fraction):

The general procedure to estimate and assess the model remains the same. I hence only pinpoint to differences and do not repeat all steps outlined for the 3PL model.

fit2PL <- mirt(d, 1, itemtype = "2PL", verbose = F)
fit2PL

#> 
#> Call:
#> mirt(data = d, model = 1, itemtype = "2PL", verbose = F)
#> 
#> Full-information item factor analysis with 1 factor(s).
#> Converged within 1e-04 tolerance after 15 EM iterations.
#> mirt version: 1.45.1 
#> M-step optimizer: BFGS 
#> EM acceleration: Ramsay 
#> Number of rectangular quadrature: 61
#> Latent density type: Gaussian 
#> 
#> Log-likelihood = -2739.232
#> Estimated parameters: 20 
#> AIC = 5518.465
#> BIC = 5602.757; SABIC = 5539.276
#> G2 (1003) = 498.96, p = 1
#> RMSEA = 0, CFI = NaN, TLI = NaN

anova(fit2PL, fit3PL)

coef(fit2PL, IRTpars = TRUE, simplify = TRUE)

#> $items
#>         a      b g u
#> V1  1.699  1.168 0 1
#> V2  1.018 -0.582 0 1
#> V3  1.084  0.343 0 1
#> V4  1.275  0.657 0 1
#> V5  1.177  0.514 0 1
#> V6  1.176 -0.146 0 1
#> V7  1.286  1.868 0 1
#> V8  0.838  0.024 0 1
#> V9  1.684  2.162 0 1
#> V10 0.919 -0.070 0 1
#> 
#> $means
#> F1 
#>  0 
#> 
#> $cov
#>    F1
#> F1  1

tracePlot(fit2PL, theta_range = c(-5, 5), facet = F, legend = T) +
  scale_color_brewer(palette = "Set3") +
  labs(title = "2PL - Traceplot")

1PL or Rasch model

The 1PL model gets rid of yet another parameter: item discrimination. It basically constrains item discrimination to be equal across all items. Only item difficulty is allowed to vary. The mathematical form hence becomes:

Note that there is no subscript for the letter , because it is constrained to be the same for all items. The Rasch model (stemming from a different scholarly tradition) is mathematically equivalent, but is often expressed slightly differently:

It is basically the same as the equation above, but the 1.7 constant is omitted. Further, the typical notational system uses instead of and instead of .

This constrained model is rather an “ideal” measurement model than a model that can be perfectly fitted to the data. Yet, if we find items that fit this model sufficiently, it actually has some mathematical properties that cannot be obtained with less constrained models (e.g., 2PL or 3PL models). For this reason, many scholars favor this measurement model (particularly those following the Rasch School of Measurement)

1. For this model, the number-correct score is particularly a sufficient estimation of theta.
2. Due to the fixed discrimination parameter, item characteristic curves cannot cross. Items thus do not differ in how well they differentiate respondents.
3. Such a model has a so-called specific objectivity: Because the Rasch model is based on the idea that items are designed to fit certain properties, instead of finding a model that fits the data, item as well as person parameter estimates are invariant across samples. Person (i.e., proficiency scores) and item parameters (i.e., item difficulty) thus will be similar even if only a subset of the item pool is used or a different population is studied. This makes it very easy to create smaller tests, parallel test or even adaptive testing pools.

fitRasch <- mirt(d, 1, itemtype = "Rasch", verbose = T)

fitRasch

#> 
#> Call:
#> mirt(data = d, model = 1, itemtype = "Rasch", verbose = T)
#> 
#> Full-information item factor analysis with 1 factor(s).
#> Converged within 1e-04 tolerance after 20 EM iterations.
#> mirt version: 1.45.1 
#> M-step optimizer: nlminb 
#> EM acceleration: Ramsay 
#> Number of rectangular quadrature: 61
#> Latent density type: Gaussian 
#> 
#> Log-likelihood = -2746.893
#> Estimated parameters: 11 
#> AIC = 5515.786
#> BIC = 5562.147; SABIC = 5527.232
#> G2 (1012) = 514.28, p = 1
#> RMSEA = 0, CFI = NaN, TLI = NaN

anova(fitRasch, fit2PL)

coef(fitRasch, IRTpars = TRUE, simplify = TRUE)

#> $items
#>     a      b g u
#> V1  1  1.673 0 1
#> V2  1 -0.618 0 1
#> V3  1  0.374 0 1
#> V4  1  0.798 0 1
#> V5  1  0.593 0 1
#> V6  1 -0.174 0 1
#> V7  1  2.304 0 1
#> V8  1  0.018 0 1
#> V9  1  3.138 0 1
#> V10 1 -0.073 0 1
#> 
#> $means
#> F1 
#>  0 
#> 
#> $cov
#>       F1
#> F1 1.302

tracePlot(fitRasch, theta_range = c(-5, 5), facet = F, legend = T) +
  scale_color_brewer(palette = "Set3") +
  labs(title = "1PL - Traceplot")

itempersonMap(fitRasch)

subset1 <- d %>%
  select(V2, V10, V3, V4, V7)
subset2 <- d %>%
  select(V6, V8, V5, V1, V9)

test1 <- mirt(subset1, 1, itemtype = "Rasch", verbose = F)
test2 <- mirt(subset2, 1, itemtype = "Rasch", verbose = F)

testInfoCompare(test1, test2)

cor.test(fscores(test1), fscores(fitRasch))

#> 
#>  Pearson's product-moment correlation
#> 
#> data:  fscores(test1) and fscores(fitRasch)
#> t = 41.294, df = 498, p-value < 2.2e-16
#> alternative hypothesis: true correlation is not equal to 0
#> 95 percent confidence interval:
#>  0.8582740 0.8981538
#> sample estimates:
#>       cor 
#> 0.8797522

cor.test(fscores(test2), fscores(fitRasch))

#> 
#>  Pearson's product-moment correlation
#> 
#> data:  fscores(test2) and fscores(fitRasch)
#> t = 38.389, df = 498, p-value < 2.2e-16
#> alternative hypothesis: true correlation is not equal to 0
#> 95 percent confidence interval:
#>  0.8405774 0.8851290
#> sample estimates:
#>      cor 
#> 0.864542

Where to go next?

IRT is a broad field and there are many more models (e.g., graded response models, multidimensional models, 4PL models…). A good starting point is the extended literature:

• DeMars, C. (2010). Item response theory. Oxford University Press.

• Embretson, S. E., & Reise, S. P. (2013). Item response theory (2nd ed.). Psychology Press.

There are also good online resources:

• Website dedicated to Rasch modeling: https://www.rasch.org/

References

• DeMars, C. (2010). Item response theory. Oxford University Press.

• Orlando, M. & Thissen, D. (2000). Likelihood-based item fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24, 50-64.

Time Management and Self-Organization

One of the best things about being an academic scholar is that it never gets boring. One has the unique privilege to engage in diverse projects and activities. Doing a PhD often means teaching some classes, participating in larger research projects, writing several papers at the same time, presenting at conferences, participating in workshops, and – on top of everything – keeping up with your actual PhD project. Time management and organisation is hence an important basis to reduce stress and frustration. I believe it is important to note that one never gets perfect at this. So don’t strive for perfection, but constantly reflect on your workflow and aim to get better instead.

1. Take notes and systematize your thoughts and projects. You will have a lot of meetings – whether with your supervisors, project partners, colleagues of fellow PhD students. Don’t think you will remember everything. Write everything down! Meetings notes, tasks, methods details, code annotations, thoughts that pop up at the most random times, summaries of important papers – you should create an archive of your thinking. Look into apps such as Evernote (my personal favorite) or OneNote. They allow to tag, search, and organize all of these notes in a meaningful manner, make them searchable, and can run on several devices.
2. Rethink your workspace and back-up your work. You think your computer files and folders are well organized? Think again. The more structure and consistency you put into organizing your digital files, the better you will find anything you want to find. Create separate folders for different aspects of your work, create further separate folders for projects and classes and find a good way to sort and browse through them. A consistent file and folder naming policy might sound a bit strict, but it can be really helpful in the long run. Most importantly, find a good way to back up your work and do it frequently (at best automatically). If your current work is not backed up, do it now!
3. Prioritize your PhD project. In the beginning, it can feel like it takes forever until you get invited to collaborative projects or until the first paper is published. Yet, stick to your project for the time being. Invitations and collaborations will eventually come and you will have more to do than you originally thought you would. Particularly in later stages of your PhD and beyond, learn to politely say no. Time management and self-organization also means to protect yourself against too much work.
4. Use the SMART approach to break down larger projects. A PhD project usually takes several years and involves several studies, which, in turn, consists of several stages. To not get overwhelmed, break your thesis down into specific, measurable, attainable, relevant and timely (SMART) goals. “Draft first paragraph” is much easier to complete than “Write chapter 1”.
5. Take breaks. Finally, I would like to suggest something seemingly paradoxical: The more you have to do, the more breaks you should take. Breaks will make you more and not less productive. They are important to recharge, to get re-motivated and simply enjoy working more. When you plan your day, don’t forget to plan breaks.

Staying Up-to-date and Organizing Your Knowledge

The following aspects are in a way still connected to self-organization, but focus a bit more on your actual scientific work. The scientific literature is growing at a fast pace. Even in subfields, one can hardly read all publications. It is hence very important to stay up-to-date and find ways to digest the large amount of information.

1. Gain an overview of the most important theories and studies in your field. You may have studied a bachelor and master program in your field, but do you really know your field? Communication Science (as any other disciplines) has become a broad discipline and it doesn’t hurt to try to get at least an idea of all of its research areas and subfields. I am not saying that you should become an expert in all of them, rather get some knowledge of the broad themes, theories, and developments. A historical understanding of the discipline is also beneficial (see this handbook for example for communication science).
2. Keep track of the literature and use a reference manager. You really need to make sure that you are aware of the latest publications in your field. Journals offer RSS feeds, google offers journal-specific alerts. Set up a system that keeps you updated with the most important journals in your field. And by all means,  use a reference manager to systematize the literature and to simplify your citing practice (my personal preference: zotero) .
3. Consider joining (some) social media platforms. This is not for everyone, but specifically Twitter (to a minor degree platform such as ResearchGate) provide you with information about new developments, access to current debates, and simply a platform to engage with other scientists. Job offers, conference calls, the newest publications (and how they are perceived…) all of this is shared on Twitter. I personally benefitted a lot from joining Twitter.
4. Engage in conversations. Don’t be shy to participate in conversations, whether in your department or on Twitter. Don’t brag, be respectful, and think before you speak, but otherwise, join the discussion! Science is about exchanging, defending and refuting ideas.
5. Participate in workshops. Conferences, institutions, departments or simply your colleagues will offer workshops on various topics. These events are not only good opportunities to learn something new, they are likewise often a lot of fun and a great way to meet new people.

Refreshing Old and Learning New Skills

You know already a lot, but obviously you don’t know everything. A PhD should also be your chance to refresh and improve some skills as well as learn entirely new methods. You will of course specialize in certain areas, but I strongly recommend to get a good overview of the existing methods as well.

1. Gain an overview of the most important methods. You think of yourself as a computational scientist and hence don’t feel like you should learn traditional survey methodology? You consider yourself an experimental researcher and don’t bother about machine learning? I guarantee that you will become a better scientist if you have a least a good understanding of what methods and areas of research exist. Never heard of eye tracking? Find out what it is about. Puzzled by people talking about Bayesian statistics? Well, don’ttry to excel at everything, but do get a grasp of what people are talking about.
2. Familiarize yourself with open science principles. Don’t wait until after your PhD to do science the right way. More and more journals adopt open science principles, tenure-track programs include open science criteria and the general science community moves towards more open and transparent research. Familiarize yourself with preregistration, data sharing possibilities, reproducibility, and open access. This paper may be a good starting point to dive into the debate and definitely get acquainted with the Open Science Framework.
3. Learn a programming language. Learning a programming language is not easy, but it is hard to get around now-a-days. Our field moves towards more and more computational methods and advanced statistics. You will want to be at the forefront of this movement. Which language you will want to learn depends on the type of research that you plan to conduct. If you are primarily interested in statistics, R might be your choice. If you want to dive deeper into machine learning, you will not get around python. If you plan to build web applications, you should look into javascript and react. This repository created by some colleagues and myself can be a great starting point for learning R for statistical analysis and text analysis. This open access book by my colleague Wouter van Atteveldt is further a great source for computational methods in both R and Python.
4. Refresh your statistic skills. Related to learning R, take the time to refresh your statistic skills. Really try to understand frequentist inferences, get to know more advanced analysis approaches such as multilevel modelling, SEM, machine learning, to name just a few. Andy Field’s book – by now almost a classic – can be a good introduction, but consider taking online courses (e.g., this coursera course by Daniel Lakens) and consult more comprehensive books (e.g., Richard McElreath’s Rethinking Statistics).
5. Explore alternatives to “standard” software. Although it might seem convenient to use a standard word processing software such as MS Word or the classic MS Powerpoint to create your presentations (and it often is!), be aware of alternatives. Latex is annoyingly techy, but has benefits in many cases (particularly for formatting a large PhD thesis). RMarkdown can offer an interesting approach to combine writing and analysis (in R) to make your work more reproducible (see this thesis by Amy Orben as an example). The book by Yihui is a great resource in this regard. Check out whether it works for you.

Doing Research

A major part of your work as PhD student is obviously doing research. You can of course simply start and see where it leads you, but I would suggest to think a bit more long-term. What type of researcher do you want to be? How do you want to be seen in the field? Don’t stress too much about these questions, but it can be beneficial to reflect a little bit on them at the start.

1. Follow the highest standards, but don’t stress yourself if things don’t work out. Always value quality over quantity. This does not mean that you should not publish and also keep publishing, but rather continuously put out good work than publish a lot of rubbish. Try to make it right the first time, but then again don’t stress if things don’t work out.
2. Get comfortable with rewriting, changing, and improving your work. It can be a bit frustrating if you just finished a draft, shared it with your colleague or supervisor, only to get it back with thousands of comments on how to improve it. Don’t be intimated or offended, instead get comfortable with rewriting and restructuring, sometimes even entirely overthrowing manuscripts, drafts or research project. The first thought is seldom gold, but nonetheless needs to be written down before one can think “the better thought”.
3. Have a clear plan for each project. Plan ahead, define milestones, schedule meetings, and keep to your plan (remember the SMART approach). It will help you easily completing the most complex projects.
4. Think about the argumentation first. Don’t just start to write. Develop a skeletal outline, sketch out your argumentation, change, restructure, fight with it, and get feedback. Only if you are happy with this outline, start to write your paper (obviously, you don’t have to be follow this rule every time, it can of course make sense to simply start writing and develop ideas simultaneously).
5. Prepare yourself for failure and frustrating review processes. Things will go wrong, but the most frustrating thing can be a seemingly unfair review process. Don’t be offended, see the merit in a reviewer’s comments, get over with the nasty bit, and move on. Simple as that (I have to admit I am not particularly good at this).

Socializing, Networking, and Becoming Part of the Community

Although some people do a PhD and do not have the goal of staying in academia, I am assuming that you want to become and stay a researcher. In order to achieve that goal, you need to grow a network. I like to emphasize that I don’t refer to networking here in the classical sense. Don’t try to meet or befriend “great scholars” for the sake of gaining advantages. Rather, I want you to have a great community of scholars that help you out if you need support, that provide you with a lively exchange of information and ideas, and generally provide the informational, instrumental, and emotional support that every person – no matter what job – needs from time to time.

1. Socialize with your lab and other colleagues, but don’t compare yourself with them. Don’t isolate yourself, but also don’t do anything that doesn’t feel right to you. Talk to colleagues, find friends, but don’t feel intimated by their success, CV, number of publications or whatever you might compare yourself on. Every career is different. Social comparison can be very detrimental and simply is not worth it.
2. Meet and exchange ideas with other PhD students. Having lively conversations and exchanges with fellow PhD students is perhaps the most important recommendation. They will understand you and provide the social environment for your academic journey. Some of you will literally grow old together.
3. Don’t struggle on your own. Don’t think that you need to figure everything out by yourself. Ask your supervisor and colleagues, be honest if you don’t know something. Don’t be shy to ask for help. A good mentor will help you tremendously.
4. Present your research in your department and at conferences. It will take some courage in the beginning, and perhaps will be a little bit scary too, but be confident in your work and present it to colleagues and other scholars on conferences. Their feedback will be valuable and they will get to know you and what you are working on.
5. Enjoy the ride together. Don’t compete, celebrate your achievements together, and have a good time. This is what this is all about.

Work-Life Balance and Mental Health

Just as any job, being an academic scholar can be very rewarding, but also be frustrating and overwhelming. Taking care of yourself is as important as having the feeling that you advance and progress. As a final advice, I would like to remind you that none of the recommendations above are requirements to succeed. You don’t need to fulfil all or any of them. Don’t be too hard on you. In the noisy confusion of life, keep true to yourself and don’t do anything that you don’t want to do.

1. Recognize your own brilliance. Many early (as well as later) career researchers suffer from imposter syndrome. They believe they may not be as competent as others perceive them to be. Don’t make yourself smaller than you are. Of course, you don’t know as much as a senior scholar, and of course you are not as proficient in R as an older, more experienced colleague, and of course you may not be as fast in writing a section of a paper than your supervisors who have done this for years. But that does not mean that you are less bright, less smart, and less qualified for this job. Be patient and trust in yourself and trust in others’ belief in you.
2. Success doesn’t come overnight. Know that you will get there eventually. Things take time and you don’t need to be faster than everybody else. In other words, be patient and beyond a healthy amount of self-discipline, be gentle with yourself.
3. Supervisors and other seniors can be and often are wrong. If you feel you are right, tell them so and make them understand why you see things differently. A comment makes no sense to you? Explain why you think so. If they request unrealistic amounts of work from you in a short period of time, tell them that this doesn’t work and that it doesn’t fit your planned schedule. They should and will have respect for your plans and schedules.
4. Enjoy and acknowledge your achievements. Take time to celebrate your achievements. You got accepted to your first conference? You should be proud! You finally got your first paper published? Celebrate with your colleagues and friends. Every step on the way is important and worth acknowledging. Take a day off and enjoy the results of your hard work.
5. If things get crazy, talk to your supervisor(s). In any job, there will come a time where things get a bit overwhelming and where you may not see the light at the end of the tunnel. Don’t be shy to tell your supervisor that you are struggling. They are there to help you out, to reduce your stress and make sure that you enjoy work again.

On a personal note…

I am immensely grateful to all the people who supported me over the last years and who helped me to get to this point in my academic career. These include teachers, mentors, colleagues, and friends in Mainz, Stuttgart, and Ithaca. I am particularly thankful to those people whose advice, mentorship, and friendship have helped me to become the researcher I am today and who helped me in planing and reflecting on my future as a communication scholar – especially in the last months:

• My parents who are and remain my most trusted mentors and advisers.
• Sabine Trepte, Christian Schemer, and Michael Scharkow whose guidance, advice, and mentorship had a profound impact on my research as well as my future plans and who helped me think clearly about what it means to be a communication scholar.
• Tobias Dienlin, Stefan Winter, German Neubaum who are not just great scholars, but true friends. Their support means all to me.
• Natalie Bazarova who not only supported my visit to Cornell, but continues to inspire and support me. I am very grateful that we have met and I am looking forward to many more collaborations in the future.
• Janis Whitlock who teaches me important life lessons in every of our conversations. I hope that our paths will cross often in the next years.
• Samuel Taylor whose friendship and collegiality means a lot to me.
• Dima Epstein and all my colleagues from the Comparative Privacy Research Network who taught me to be patient while remaining immensely supportive.
• I also want to thank all my colleagues in Hohenheim, Ithaca, and Mainz for making my day-to-day job an inspiring and fun endeavor.
• Maria, for her unconditional support and love.

Take home message

A highly specific antibody test (e.g., Pr(tests−|is−) = 0.95) does not mean that the probability of being correctly identified as “positive” (i.e, Pr(is+|tests+)) is also high. In simple words, a large amount of people (the majority) who will get a positive test results are in fact not immune.

Footnotes

How does exponential growth actually look like?

In what follows, I define a small function that allows us to simulate an exponentially growing vector. This means that each value is the previous value multiplied by a certain factor:

Before we simulate this in R, play with it: change the daily growth rate and press Play. Then switch to the log axis, which is the trick behind the famous FT chart.

library(tidyverse)
library(papaja)

# Exponential function
exp_growth <- function(start = 1, 
                  steps = 10, 
                  factor = 2) {
  x <- vector()
  x[1] <- start
  for(i in 2:steps){
    x[i] = x[i-1]*factor
  }
  return(x)
}

# Where are we after only 10 steps?
exp_growth()

#>  [1]   1   2   4   8  16  32  64 128 256 512

If we run the function with the default values (starting value = 1, steps = 10, factor = 2), we reach the number 512 after only 10 steps. Impressive growth, right?

# Where are we after 20 steps?
exp_growth(steps = 20)

#>  [1]      1      2      4      8     16     32     64    128    256    512
#> [11]   1024   2048   4096   8192  16384  32768  65536 131072 262144 524288

After only 20 steps, however, we already reach a number that is larger than 0.5 million.

To get a true understanding of exponential growth, let us visualize 30 steps (while still using a starting value of 1).

(plot1 <- data.frame(n = 1:30,
                     value = exp_growth(steps = 30)) %>%
  ggplot(aes(x = n, y = value)) +
  geom_point() +
  geom_line() +
  theme_bw())

After 30 steps only, we have reached the crazy large number of 536,870,912 (more than 500 million!). At the same time, it seems that the growth only really takes off after 25 steps. But remember our previous calculation. We were already at 0.4 million after 20 steps. So let’s zoom in on the first 10 steps again:

plot1 +
  ylim(0,750) +
  scale_x_continuous(breaks = c(1:10), limits = c(1,10))

After only 10 steps, we reached a number higher than 500. This steep growth was buried in the previous figure because the growth afterwards was just sooo much steeper.

Now what happens if we logarithmize the y-axis (as it was done in the figure referenced above)?

plot1 +
  scale_y_log10()

With only one short change in the code, it seems like we have plotted a linear growth. But again, check the y-axis closely. The distance between two points on the axis is not as equal as the figure suggest. From step 1 to step 10, the difference is 1:512 (an increase of 511). However, from step 10 to step 20, 512:524,288 (an increase of 523,776!).

Exponential growth based on percentages

Now in the case of the Corona virus spreading, we often quantify growth in percentage. So let’s consider another example. I now define a function that let’s us investigate different growth rates (based on percentages). So we again simulate a vector that represent exponential growth. This time, however, each value is the previous value plus a certain percentage of the previous value:

perc_growth <- function(start = 1, steps, perc) {
  x <- vector()
  x[1] <- start
  for(i in 2:steps){
    x[i] = x[i-1]+ perc*x[i-1]
  }
  return(x)
}

# Recreating the exponential growth from above
perc_growth(start = 1, steps = 10, perc = 1)

#>  [1]   1   2   4   8  16  32  64 128 256 512

We see that if we start with 1 case and expect a 100% growth each day, we end up with 512 cases after 10 steps (e.g., 10 days). This is the exact exponential growth that we modelled above.

Of course, growth is not always 100%. It can be less, of course (but also more, like 200%!). In the next example, we see the effect of 5%, 10% and 20% growth on the growth curve.

(plot2 <- data.frame (n = 1:30,
            "5" = perc_growth(start = 1, steps = 30, perc = 0.5),
            "10" = perc_growth(start = 1, steps = 30, perc = 0.10),
            "20" = perc_growth(start = 1, steps = 30, perc = 0.20)) %>%
  gather(key, value, -n) %>%
  ggplot(aes(x = n, y = value, color = key)) +
  geom_point() +
  geom_line() +
  theme_bw())

Again we see that the growth of 50% is a lot steeper compared to 5% or 10%. Let’s again zoom in.

plot2 +
  xlim(0, 10) +
  ylim(0, 50)

As we can see, a 50% growth leads to ~40 cases after 10 steps (bear in mind, we started with only 1).

Growth curves of the corona virus

Now let us visualize the growth curves of the corona virus infections per country. I therefore use the public data provided by the John Hopkins University (https://github.com/CSSEGISandData/COVID-19). For this purpose, I am only looking at the data from Germany.

# JHU CSSE data as of 12 March 2020 (pinned to that day's commit)
url <- paste0("https://raw.githubusercontent.com/CSSEGISandData/COVID-19/",
              "82e1397911272be823cb93cbc19b62551608b8dc/",
              "csse_covid_19_data/csse_covid_19_time_series/time_series_19-covid-Confirmed.csv")

d <- read.csv(url) %>%
  select(-Lat, -Long)

d <- d %>%
  rename(country = Country.Region) %>%
  filter(country == "Germany") %>%
  gather(date, value, -country) %>%
  separate(date, c("x", "date"), sep = 1) %>%
  mutate(cases = as.numeric(value),
         date = lubridate::mdy(date)) %>%
  as_tibble

# plot
(plot3 <- d %>%
  ggplot(aes(x = date, y = cases)) +
  geom_point() +
  geom_line() +
  theme_bw() +
  labs(x = "date", y = "cases"))

If we compare this figure to the exponential growth curves that we simulated above, it should become clear that the actual growth curve of the corona pandemic in Germany is likewise exponential.

Again, if we log-transformed the y-axis and stretch the width of the figure (as shown in the tweet above), it again looks very different.

d %>%
  ggplot(aes(x = date, y = cases + 1)) +
  geom_point() +
  geom_smooth(se = F, color = "black") +
  theme_bw() +
  scale_y_log10() +
  labs(x = "date", y = "cases")

But what is the actual growth rate (in percentage)? And what do we need to expect in the near future?

d %>%
  mutate(cases_day = cases - lag(cases),
         percent = cases_day/cases) %>%
  summarize(growthrate = mean(percent, na.rm = T),
            se = psych::describe(percent)$se,
            lower = growthrate - 1.96*se,
            upper = growthrate + 1.96*se)

#> # A tibble: 1 × 4
#>   growthrate     se  lower upper
#>        <dbl>  <dbl>  <dbl> <dbl>
#> 1      0.154 0.0318 0.0918 0.216

The growth curve is on average ~15% (of course, we can debate if simply averaging across the growth rates per day is a good approach. Actually the figure in the tweet referenced above features a dotted line that represents a daily growth rate of 33% and most countries (including Germany) seem to approach that line). If we nonetheless use a growth rate of 15%, how would the increase in cases look like in a simulation for the next weeks and months?

Our starting value is 3,156 (the number of registered infections in Germany today).

data.frame(n = 1:30,
           cases = perc_growth(start = 3156, steps = 30, perc = 0.154),
           lower = perc_growth(start = 3156, steps = 30, perc = 0.0918),
           upper = perc_growth(start = 3156, steps = 30, perc = 0.216)) %>%
  gather(key, cases, -n) %>%
  ggplot(aes(x = n, y = cases, color = key)) +
  geom_line() +
  theme_bw() +
  ylim(0, 200000) +
  scale_color_manual(values=c("black", "grey", "grey")) +
  scale_x_continuous(breaks = seq(0,30, by = 5)) +
  labs(x = "days", title = "Simulated growth in Germany")

The grey lines show the uncertainty in the simulation. The black curve shows the average growth rate based on the available data. It reveals that if the trend would continue, we would have more than 100,000 cases after only 25 days. Yet there are several things that have to be taken into account: First, the growth rate of 15% might be wrong (I admit that averaging across all days is a very crude way of estimating the growth rate) or may change over time. The large uncertainty (grey lines) shows already how largely the actual growth rate could vary based on my crude measure. For example, if the growth rate would be ~9%, we would have ~25,000 cases instead of ~100,000 after 25 days.

So how come that the news talk about a 33% growth curve now? Well, they only started to compute the growth rate after ~100 cases were identified. In the following figure, you see a simulated 33% growth curve plotted against the actual growth curve from the day when Germany had 100 identified cases.

(plot4 <- d %>%
  filter(cases >= 100) %>%
  mutate("cases of corona" = cases,
         "33% growth curve" = perc_growth(start = 130, steps = 11, perc = .33)) %>%
  select(date, "cases of corona", "33% growth curve") %>%
  gather(key, value, -date) %>%
  ggplot(aes(x = date, y = value, color = key)) +
  geom_point() +
  geom_line() +
  theme_bw() +
  scale_color_manual(values = c("grey", "black")) +
  labs(x = "date", y = "cases", color = ""))

As you can see, the curves align pretty well. If the daily growth rate would indeed approach 33% and stay this high for the next weeks, we would have ~2,961,940 cases after 25 days.

This is why acting now (both individually and on a societal level) is so important. We need to slow down this growth considerably. But we not only have to lower the actual infections, but also have to reduce the number of people who are infected at the same time. The latter is particularly important as the health system would otherwise not be able to treat all infected with the same care or even at all. The following video illustrates the effect of less infections at the same time on the health system very well:

#Corona breitet sich aus. Durch unser Verhalten können wir alle dazu beitragen, seine Verbreitung einzudämmen. Es geht darum, die Älteren und Schwachen zu schützen und unser Gesundheitssystem nicht zu überlasten. #flattenthecurve! #einHerzfüreinander pic.twitter.com/SrtK3DSQnB

— Paul Ziemiak (@PaulZiemiak) March 14, 2020

Conclusion

Understanding exponential growth is hard. The problem is that we constantly underestimate the actual growth and its implications. In case of infections, this is very worrisome. And I fear that figures that log-transformed the y-axis (such as the one presented above) might provide a false picture of the actual growth.

So what do we need to do? With no treatment and a viable vaccine yet available, the only effective way to keep the coronavirus pandemic at bay is to give the virus less chances of spreading. The following list of actions are recommended to serve as a set of guidelines:

1. Don’t panic, but be alert.
2. Wash your hands often and practice good cough and sneeze etiquette.
3. Try to touch your face as little as possible, including your mouth, nose, and eyes.
4. Practice social distancing, no hugs and kisses, no handshakes, no high fives. If you must, use safer alternatives.
5. Do not attend concerts, stage plays, sporting events, or any other mass entertainment events.
6. Refrain from visiting museums, exhibitions, movie theaters, night clubs, and other entertainment venues.
7. Stay away from social gatherings and events, like club meetings, religious services, and private parties.
8. Reduce your amount of travel to a minimum. Don’t travel long distances if not absolutely necessary.
9. Do not use public transportation if not absolutely necessary.
10. If you can work from home, work from home. Urge your employer to allow remote work if needed.
11. Replace as many social interactions as possible with remote alternatives like phone calls or video chat.
12. Do not leave your home if not absolutely necessary.

Be safe!

Power for a bivariate relationship with manifest variables

Let us first consider a simple bivariate relationship between two manifest variables. In other words, we investigate the power for a relationship between two variables with “no measurement error”. The approach that I am going to use throughout this post is always as follows:

1. Define a function that simulates data, draw samples, estimate the relevant models, and extract parameters of interest.
2. Use grid_search() to run the simulations.
3. Count the number of significant results to “get” power.
4. Visualize the power curve.

The first step requires defining a function with the following arguments: a) simNum = the iteration number (do not change this argument, it is required by grid_search()), b) n = the sample sizes that should be tests, c) any other variables that should vary across the simulations (could be effect size(s) of the relationship(s) of interest or factor loadings of latent variables, etc.). Within this function, we use the function gen_data() that generates samples from the predefined data properties.
These properties include a “factor matrix” and an “effect matrix”. Defining both matrixes requires a good knowledge about the model that one wants to estimate. All steps are outlined below.

# Define the function
model_bivar <- function(simNum, n, b1) {
  
  # compute factor matrix 
  factor <- matrix(c(1, 0,    # x (no measurement error)
                     0, 1),   # y (no measurement error)
                   nrow = 2, ncol = 2, 
                   byrow = TRUE, 
                   dimnames = list(c('x', 'y'),  # rows
                                   c('x', 'y'))) # cols 
  # compute effect matrix
  y_res <- sqrt(1 - b1^2) 
  effect <- matrix(c(1, b1,    # b1 = effect size of the relationship
                     0, y_res), 
                    nrow = 2, ncol = 2, 
                    byrow = TRUE, 
                    dimnames = list(c('x', 'y'),    # rows
                                    c('x', 'y')))   # columns
  
  # generate data based on both matrices
  data <- paramtest::gen_data(factor, effect, n_cases = n)
  
  # fit lavaan model
  model <- '
    y ~ b1*x
  '
  fit <- lavaan::sem(model, data = data)
  
  # extract relevant parameters
  ests <- subset(lavaan::parameterEstimates(fit), label == "b1") 
  return(ests)
}

Now, we use grid_search() to run the simulations. In this case, we investigate how many participants we need to sample in order to test a bivariate effect between manifest variables of = .10 (small according to Cohen) to reach different levels of power.

We hence provide a list with the varying parameters (in this case only the sample sizes). It is important to run enough iterations. As always, results stabilize with higher iterations. Side note: We can profit from parallel computing by using parallel = 'snow' and ncpus = [number of available cores]. This increases the computation speed considerably.

# Run simulations (L'Ecuyer-CMRG makes the parallel random numbers reproducible)
RNGkind("L'Ecuyer-CMRG")
set.seed(2020)
power_bivar <- cached("sims/power_bivar.rds", grid_search(
  func = model_bivar,
  params = list(n = seq(100, 1500, by = 50)),
  b1 = .1,
  n.iter = 1000,
  output = 'data.frame',
  parallel = 'snow',
  ncpus = parallel::detectCores() - 1
  ))

# result table
results(power_bivar) %>% as_tibble

#> # A tibble: 29,000 × 12
#>     iter n.test lhs   op    rhs   label     est     se      z pvalue ci.lower
#>    <int>  <dbl> <chr> <chr> <chr> <chr>   <dbl>  <dbl>  <dbl>  <dbl>    <dbl>
#>  1     1    100 y     ~     x     b1     0.0640 0.0990  0.646 0.518  -0.130  
#>  2     2    100 y     ~     x     b1     0.160  0.0994  1.61  0.107  -0.0345 
#>  3     3    100 y     ~     x     b1     0.0697 0.0970  0.718 0.473  -0.121  
#>  4     4    100 y     ~     x     b1     0.175  0.106   1.65  0.0998 -0.0334 
#>  5     5    100 y     ~     x     b1    -0.0444 0.0952 -0.467 0.641  -0.231  
#>  6     6    100 y     ~     x     b1     0.182  0.106   1.72  0.0851 -0.0251 
#>  7     7    100 y     ~     x     b1     0.184  0.0907  2.03  0.0425  0.00621
#>  8     8    100 y     ~     x     b1     0.0676 0.110   0.616 0.538  -0.147  
#>  9     9    100 y     ~     x     b1     0.115  0.102   1.12  0.261  -0.0856 
#> 10    10    100 y     ~     x     b1    -0.0108 0.0941 -0.115 0.908  -0.195  
#> # ℹ 28,990 more rows
#> # ℹ 1 more variable: ci.upper <dbl>

The resulting data frame includes relevant parameters for the relationship between x and y. The column n.test indicates the sample size of each iteration.

To compute the power, we only need to count the number of significant results per sample size group (n = 100, 150, 200, 250,…). We can then check the “needed” sample size for various power levels. It would also suggest plotting a power curve as it provides more information than a single value alone.

# compute power
(result_bivar <- results(power_bivar) %>%
  group_by(n.test) %>%
  mutate(sig = (pvalue < .05)) %>%
  summarise(power = mean(sig, na.rm=TRUE)))

#> # A tibble: 29 × 2
#>    n.test power
#>     <dbl> <dbl>
#>  1    100 0.171
#>  2    150 0.235
#>  3    200 0.311
#>  4    250 0.337
#>  5    300 0.415
#>  6    350 0.462
#>  7    400 0.552
#>  8    450 0.58 
#>  9    500 0.614
#> 10    550 0.664
#> # ℹ 19 more rows

# Extract sample size for 80% power
result_bivar %>%
  filter(power >= 0.80)

#> # A tibble: 15 × 2
#>    n.test power
#>     <dbl> <dbl>
#>  1    800 0.802
#>  2    850 0.826
#>  3    900 0.866
#>  4    950 0.872
#>  5   1000 0.887
#>  6   1050 0.895
#>  7   1100 0.921
#>  8   1150 0.941
#>  9   1200 0.942
#> 10   1250 0.944
#> 11   1300 0.959
#> 12   1350 0.963
#> 13   1400 0.96 
#> 14   1450 0.97 
#> 15   1500 0.979

To plot a power curve, I add two things to the raw proportions: a 95% confidence interval for each estimate (with 1,000 iterations, the Monte Carlo error is about ±3 percentage points) and a smooth curve. Because power is approximately , a probit regression on fits these curves very well, and we can solve it for the sample size that reaches 80% power.

# Power with Monte Carlo intervals, fitted curve and the n for 80% power
pw_bivar  <- summarise_power(results(power_bivar), pvalue < .05)
fit_bivar <- power_fit(pw_bivar)
n80_bivar <- n_for_power(fit_bivar)

ggplot(pw_bivar, aes(n, power)) +
  geom_hline(yintercept = .80, linetype = "dashed", colour = pal["grey"]) +
  annotate("segment", x = n80_bivar, xend = n80_bivar, y = 0, yend = .80,
           linetype = "dotted", colour = pal["grey"]) +
  geom_line(data = power_curve(fit_bivar, 100:1500), colour = pal["blue"], linewidth = 1) +
  geom_pointrange(aes(ymin = lower, ymax = upper), colour = pal["blue"], size = .2) +
  annotate("text", x = n80_bivar + 30, y = .74, hjust = 0, colour = "#7a7f88",
           label = paste0("80% power at n ≈ ", fmt_n(n80_bivar))) +
  scale_y_continuous(labels = scales::percent, limits = c(0, 1)) +
  labs(x = "Sample size", y = "Power") +
  theme_blog()

In this example, we see that we need roughly 770 participants in order to test the small correlation between the two variables with a power of 80% (assuming an alpha-level of .05!).

For this simple example, this procedure is a bit complex as a solution can also be derived analytically (any remaining difference in the resulting sample size is explainable by the Monte Carlo error of our simulations. With more iterations, we would get even closer to the analytical solution):

library(pwr)
pwr.r.test(r = c(.1),
           sig.level = .05,
           power = .80,
           alternative = "two.sided")

#> 
#>      approximate correlation power calculation (arctangh transformation) 
#> 
#>               n = 781.7516
#>               r = 0.1
#>       sig.level = 0.05
#>           power = 0.8
#>     alternative = two.sided

Power for a bivariate relationship between latent variables

The whole procedure becomes more complex if we try to simulate the power for a relationship between latent variables. Here, we need to make assumptions about the items’ reliability. In this first example, we assume that we have comparatively good scales whose items have factor loadings of .70 (i.e., each item’s reliability is .70² = .49, and the reliability of a three-item scale is ω ≈ .74). Defining the factor matrix is hence more complex and the model is now a structural equation model with latent variables. Otherwise, the procedure remains the same.

# define function
model_sem <- function(simNum,
                      n, 
                      b1, 
                      x_ind1, x_ind2, x_ind3, 
                      y_ind1, y_ind2, y_ind3) {
  
  # compute factor matrix (in this case with measurement error)
  factor <- matrix(c(x_ind1, 0,   # y1
                     x_ind2, 0,   # y2
                     x_ind3, 0,   # y3
                     0, y_ind1,   # y1
                     0, y_ind2,   # y2
                     0, y_ind3),  # y3
                   nrow=6, ncol=2, 
                   byrow=TRUE, 
                   dimnames=list(c('x1', 'x2', 'x3', 'y1', 'y2', 'y3'),  # rows (observed)
                                 c('x', 'y')))                           # columns (latent)
  
  # matrix of the effects structure (variance-covariance)
  y_res <- sqrt(1 - b1^2)
  effect <- matrix(c(1, b1,     # x
                     0, y_res), # y
                    nrow=2, ncol=2, 
                    byrow=TRUE, 
                    dimnames=list(c('x', 'y'),    # rows
                                  c('x', 'y')))   # columns
  
  # generate data based on both matrices
  data <- paramtest::gen_data(factor, effect, n_cases = n)

  # fit lavaan model
  model <- '
   # measurement model
   x =~ x1 + x2 + x3
   y =~ y1 + y2 + y3

   # structural regression model
   y ~ b1*x
  '
  fit <- lavaan::sem(model, data = data)
  ests <- lavaan::parameterEstimates(fit)
  
  # extract relevant parameters
  est <- ests[ests$label == 'b1', 'est']
  p <- ests[ests$label == 'b1', 'pvalue']
  return(c(est = est, 
           p = p, 
           sig = (p < .05)))
}

# Measurement error specified by factor loading = .7
RNGkind("L'Ecuyer-CMRG")
set.seed(2021)
power_sem1 <- cached("sims/power_sem1.rds", grid_search(
  func = model_sem, 
  params = list(n = seq(500, 4000, by = 100)),
  b1 = .1,
  n.iter = 1000, 
  output = 'data.frame',
  x_ind1 = .7,    
  x_ind2 = .7, 
  x_ind3 = .7,      
  y_ind1 = .7, 
  y_ind2 = .7, 
  y_ind3 = .7, 
  parallel = 'snow',
  ncpus = parallel::detectCores() - 1
  ))

Let us quickly check the resulting data frame.

results(power_sem1) %>% as_tibble

#> # A tibble: 36,000 × 5
#>     iter n.test     est       p   sig
#>    <int>  <dbl>   <dbl>   <dbl> <dbl>
#>  1     1    500 0.00710 0.906       0
#>  2     2    500 0.182   0.0111      1
#>  3     3    500 0.165   0.00259     1
#>  4     4    500 0.0111  0.866       0
#>  5     5    500 0.142   0.00922     1
#>  6     6    500 0.111   0.110       0
#>  7     7    500 0.0879  0.168       0
#>  8     8    500 0.169   0.0112      1
#>  9     9    500 0.109   0.0469      1
#> 10    10    500 0.126   0.0396      1
#> # ℹ 35,990 more rows

This time, we only extracted the estimate, the p-value, and already transformed the p-value into a binary variable (p < .05). Replications in which the model does not converge return NA and are dropped when we compute power (na.rm = TRUE).

But before we estimate the power, let us quickly run the simulation again, but change the reliability of the items to more realistic values (at least in the social sciences, scales often have items with factor loadings around .50). This is very easy, as the defined function has arguments for each factor loading in the model.

We then combine the results from both simulations. After a bit of data wrangling, we can now plot both power curves.

# Let's also test a more realistic example (factor loadings = .5)
set.seed(2022)
power_sem2 <- cached("sims/power_sem2.rds", grid_search(
  func = model_sem, 
  params = list(n = seq(500, 4000, by = 100)),
  b1 = .1,
  n.iter = 1000, 
  output = 'data.frame',
  x_ind1 = .5,    # changed to .5 
  x_ind2 = .5, 
  x_ind3 = .5,      
  y_ind1 = .5, 
  y_ind2 = .5, 
  y_ind3 = .5, 
  parallel = 'snow',
  ncpus = parallel::detectCores() - 1
  ))

# Extract result table
(result_sem1 <- results(power_sem1) %>%
    mutate(rel = .7) %>%
    bind_rows(results(power_sem2) %>%
                mutate(rel = .5)) %>%
  group_by(rel, n.test) %>%
  summarise(power=mean(sig, na.rm=TRUE)))

#> # A tibble: 72 × 3
#> # Groups:   rel [2]
#>      rel n.test power
#>    <dbl>  <dbl> <dbl>
#>  1   0.5    500 0.175
#>  2   0.5    600 0.192
#>  3   0.5    700 0.232
#>  4   0.5    800 0.3  
#>  5   0.5    900 0.301
#>  6   0.5   1000 0.346
#>  7   0.5   1100 0.363
#>  8   0.5   1200 0.4  
#>  9   0.5   1300 0.419
#> 10   0.5   1400 0.441
#> # ℹ 62 more rows

pw_sem <- bind_rows(
  summarise_power(results(power_sem1), sig == 1) |> mutate(loadings = ".70"),
  summarise_power(results(power_sem2), sig == 1) |> mutate(loadings = ".50"))
fits_sem <- lapply(split(pw_sem, pw_sem$loadings), power_fit)
n80_sem  <- sapply(fits_sem, n_for_power)
curves_sem <- bind_rows(lapply(names(fits_sem), \(l) power_curve(fits_sem[[l]], 500:4000) |> mutate(loadings = l)))

ggplot(pw_sem, aes(n, power, colour = loadings)) +
  geom_hline(yintercept = .80, linetype = "dashed", colour = pal["grey"]) +
  geom_line(data = curves_sem, linewidth = 1) +
  geom_pointrange(aes(ymin = lower, ymax = upper), size = .2) +
  annotate("text", x = 4000, y = c(.20, .12), hjust = 1, colour = pal[c("blue", "orange")],
           label = c("loadings .70 (ω ≈ .74)", "loadings .50 (ω = .50)")) +
  scale_colour_manual(values = c(".70" = pal[["blue"]], ".50" = pal[["orange"]])) +
  scale_y_continuous(labels = scales::percent, limits = c(0, 1)) +
  labs(x = "Sample size", y = "Power") +
  theme_blog()

We can see two very interesting things in this plot: First, a small correlation between two latent variables requires a LOT more participants than the same correlation between manifest variables (of course: There is no measurement error in the model). But the difference is considerable depending on the items’ reliability. Whereas factor loadings of .70 already require more participants to reach a power of 80% (roughly 1,420 compared to 770 in the first example), reducing factor loadings to .50 (the more realistic scenario; the scale reliability then drops to ω = .50) requires about 3,200 (!!!) participants to reach a power of 80%.

Power for a simple mediation model with latent variables

At last, let us consider a simple mediation model with latent variables. Again, the procedure remains the same, only the factor and effect matrix needs to be defined correctly (I again assumed good scales with factor loadings of .70). Likewise, the SEM needs to be specified correctly and the parameters of interest need to be extracted. Interestingly, we can also compute the power to detect the indirect or total effect even though we may only have assumptions about the direct paths in the model!

model_med <- function(simNum,
                      n, 
                      b1, 
                      b2,
                      b3,
                      x_ind1, x_ind2, x_ind3,
                      m_ind1, m_ind2, 
                      y_ind1, y_ind2, y_ind3) {
  
  # compute factor matrix (in this case with measurement error)
  factor <- matrix(c(x_ind1, 0, 0,   # x1
                     x_ind2, 0, 0,   # x2
                     x_ind3, 0, 0,   # x3
                     0, m_ind1, 0,   # m1
                     0, m_ind2, 0,   # m2
                     0, 0, y_ind1,   # y1
                     0, 0, y_ind2,   # y2
                     0, 0, y_ind3),  # y3
                   nrow=8, ncol=3, 
                   byrow=TRUE, 
                   dimnames=list(c('x1', 'x2', 'x3', 'm1', 'm2', 'y1', 'y2', 'y3'),  # rows (observed)
                                 c('x', 'm', 'y')))                           # columns (latent)
  
  # matrix of the effects structure: gen_data() computes latent = Z %*% effect,
  # with Z independent standard normal. To get m = b1*x + e_m and
  # y = b2*x + b3*m + e_y (all latent variances = 1), the weights must be:
  m_res <- sqrt(1 - b1^2)          # m's own (residual) part
  y_x   <- b2 + b1 * b3            # weight of x in y = direct + indirect effect
  y_m   <- b3 * m_res              # weight of m's residual part in y
  y_res <- sqrt(1 - y_x^2 - y_m^2) # y's own (residual) part
  effect <- matrix(c(1, b1,    y_x,   # x
                     0, m_res, y_m,   # m
                     0, 0,     y_res),# y
                    nrow=3, ncol=3, 
                    byrow=TRUE, 
                    dimnames=list(c('x', 'm', 'y'),    # rows
                                  c('x', 'm', 'y')))   # columns
  
  # generate data based on both matrices
  data <- paramtest::gen_data(factor, effect, n_cases = n)

  # fit lavaan model
  model <- '
   # measurement model
   x =~ x1 + x2 + x3
   y =~ y1 + y2 + y3
   m =~ m1 + m2

   # structural regression model
   m ~ b1*x
   y ~ b2*x + b3*m

   ind := b1*b3
   total := b2 + ind
  '
  fit <- lavaan::sem(model, data = data)
  ests <- lavaan::parameterEstimates(fit)
  
  # extract relevant parameters
  b1_est     <- ests[ests$label == 'b1', 'est']
  b1_p       <- ests[ests$label == 'b1', 'pvalue']
  b2_est     <- ests[ests$label == 'b2', 'est']
  b2_p       <- ests[ests$label == 'b2', 'pvalue']
  b3_est     <- ests[ests$label == 'b3', 'est']
  b3_p       <- ests[ests$label == 'b3', 'pvalue']
  ind_est    <- ests[ests$label == 'ind', 'est']
  ind_p      <- ests[ests$label == 'ind', 'pvalue']
  total_est  <- ests[ests$label == 'total', 'est']
  total_p    <- ests[ests$label == 'total', 'pvalue']

  return(c(b1_est   = b1_est,    b1_p    = b1_p,
           b2_est   = b2_est,    b2_p    = b2_p,
           b3_est   = b3_est,    b3_p    = b3_p,
           ind_est  = ind_est,   ind_p   = ind_p,
           total_est= total_est, total_p = total_p))
}

RNGkind("L'Ecuyer-CMRG")
set.seed(2023)
power_med <- cached("sims/power_med.rds", grid_search(
  func = model_med, 
  params = list(n = seq(700, 4500, by = 100)),
  b1 = .1,  # assuming a small effect for all three paths       
  b2 = .1,
  b3 = .1,
  n.iter = 1000, 
  output = 'data.frame',
  x_ind1 = .7,    
  x_ind2 = .7, 
  x_ind3 = .7, 
  m_ind1 = .7,    
  m_ind2 = .7,
  y_ind1 = .7, 
  y_ind2 = .7, 
  y_ind3 = .7, 
  parallel = 'snow',
  ncpus = parallel::detectCores() - 1
  ))

# Extract result table
(result_med <- results(power_med) %>%
  group_by(n.test) %>%
    mutate(b1_sig = (b1_p < .05),
           b2_sig = (b2_p < .05),
           b3_sig = (b3_p < .05),
           ind_sig = (ind_p < .05),
           total_sig = (total_p < .05)) %>%
  summarise(b1_power = mean(b1_sig, na.rm=TRUE),
            b2_power = mean(b2_sig, na.rm=TRUE),
            b3_power = mean(b3_sig, na.rm=TRUE),
            ind_power = mean(ind_sig, na.rm=TRUE),
            total_power = mean(total_sig, na.rm=TRUE)))

#> # A tibble: 39 × 6
#>    n.test b1_power b2_power b3_power ind_power total_power
#>     <dbl>    <dbl>    <dbl>    <dbl>     <dbl>       <dbl>
#>  1    700    0.401    0.505    0.405    0.0557       0.589
#>  2    800    0.423    0.564    0.404    0.0652       0.637
#>  3    900    0.485    0.592    0.489    0.0956       0.666
#>  4   1000    0.550    0.662    0.530    0.145        0.756
#>  5   1100    0.561    0.694    0.557    0.148        0.767
#>  6   1200    0.599    0.727    0.598    0.195        0.819
#>  7   1300    0.648    0.746    0.662    0.246        0.825
#>  8   1400    0.693    0.781    0.690    0.287        0.851
#>  9   1500    0.698    0.829    0.712    0.309        0.879
#> 10   1600    0.731    0.811    0.736    0.378        0.882
#> # ℹ 29 more rows

pw_med <- results(power_med) |>
  pivot_longer(ends_with("_p"), names_to = "effect", values_to = "p") |>
  mutate(effect = sub("_p$", "", effect)) |>
  group_by(effect) |>
  group_modify(~ summarise_power(.x, p < .05)) |>
  ungroup()
fits_med <- lapply(split(pw_med, pw_med$effect), power_fit)
n80_med  <- sapply(fits_med, n_for_power)
curves_med <- bind_rows(lapply(names(fits_med), \(e) power_curve(fits_med[[e]], 700:4500) |> mutate(effect = e)))

# Three direct paths in one colour, the indirect and total effect highlighted
type <- c(b1 = "direct", b2 = "direct", b3 = "direct", ind = "indirect", total = "total")
cols <- c(direct = pal[["blue"]], indirect = pal[["orange"]], total = pal[["green"]])
labs_med <- tibble(type = c("direct", "total", "indirect"),
                   label = c("direct paths a, b, c′", "total effect c′ + a × b", "indirect effect a × b"),
                   y = c(.30, .22, .14))

ggplot(mutate(pw_med, type = type[effect]), aes(n, power, colour = type, group = effect)) +
  geom_hline(yintercept = .80, linetype = "dashed", colour = pal["grey"]) +
  geom_line(data = mutate(curves_med, type = type[effect]), linewidth = .9) +
  geom_pointrange(aes(ymin = lower, ymax = upper), size = .15, alpha = .6) +
  geom_text(data = labs_med, aes(x = 4500, y = y, label = label, colour = type),
            inherit.aes = FALSE, hjust = 1) +
  scale_colour_manual(values = cols) +
  scale_y_continuous(labels = scales::percent, limits = c(0, 1)) +
  labs(x = "Sample size", y = "Power") +
  theme_blog()

The result shows (as expected) that we need roughly the same sample size (about 1,440 to 1,810) to test the three direct paths in the model (as we assumed the same effect size of .10 for all three relationships). The path from x to m reaches 80% a bit earlier than the two paths into y, because the latter are estimated simultaneously (x and m are correlated) and the mediator is measured with only two indicators. However, we need considerably more participants to test the indirect effect (.10 × .10 = .01) with the same power! Here, we would need roughly 2,730 participants! Note that lavaan tests the indirect effect with the delta method (a Sobel-type test), which is known to be conservative. Bootstrapped or Monte Carlo confidence intervals have somewhat more power, but they are too slow to compute in every replication of a simulation like this one.

Conclusion

Simulating power for complex models is always going to be challenging due to the large number of parameters that one needs to make assumptions about. That said, the paramtest package provides a framework for doing so. In the examples above, I only varied the sample sizes. But I could have varied the effect sizes (or any other parameter in the simulation), too. This would allow analyzing the interaction between sample size and effect size on power to test such effects – from my point of view, this should be an important step before any study that aims at implementing structural equation modeling.

sessionInfo()

#> R version 4.4.0 (2024-04-24)
#> Platform: aarch64-apple-darwin20
#> Running under: macOS 26.6.2
#> 
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib 
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
#> 
#> Random number generation:
#>  RNG:     L'Ecuyer-CMRG 
#>  Normal:  Inversion 
#>  Sample:  Rejection 
#>  
#> locale:
#> [1] C.UTF-8/C.UTF-8/C.UTF-8/C/C.UTF-8/C.UTF-8
#> 
#> time zone: Europe/Amsterdam
#> tzcode source: internal
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#>  [1] pwr_1.3-0       paramtest_0.1.0 lubridate_1.9.3 forcats_1.0.0  
#>  [5] stringr_1.6.0   dplyr_1.2.0     purrr_1.2.1     readr_2.1.5    
#>  [9] tidyr_1.3.2     tibble_3.3.1    ggplot2_4.0.2   tidyverse_2.0.0
#> 
#> loaded via a namespace (and not attached):
#>  [1] gtable_0.3.6       jsonlite_2.0.0     compiler_4.4.0     tidyselect_1.2.1  
#>  [5] scales_1.4.0       yaml_2.3.12        fastmap_1.2.0      R6_2.6.1          
#>  [9] labeling_0.4.3     generics_0.1.4     knitr_1.51         htmlwidgets_1.6.4 
#> [13] pillar_1.11.1      RColorBrewer_1.1-3 tzdb_0.4.0         rlang_1.1.7       
#> [17] utf8_1.2.6         stringi_1.8.7      xfun_0.57          S7_0.2.1          
#> [21] timechange_0.3.0   cli_3.6.5          withr_3.0.2        magrittr_2.0.4    
#> [25] digest_0.6.39      grid_4.4.0         hms_1.1.3          lifecycle_1.0.5   
#> [29] vctrs_0.7.2        evaluate_1.0.5     glue_1.8.0         farver_2.1.2      
#> [33] rmarkdown_2.30     tools_4.4.0        pkgconfig_2.0.3    htmltools_0.5.8.1

The idea behind specr

So far, most researchers used loop-based functions in R to estimate all the models. They set up a results frame which gets filled with the results from each specification through a sequential analysis of all models. Although this makes a lot of sense, it has some limitations such as slow computation (due to the sequential nature) or a somewhat complex coding procedure.

Withspecr, we wanted to create convenient functions that standardize this procedure and allow for faster computation. Although not implement yet, this approach will allow to use parallelization of the model estimations. We are currently investigating the implementation of the great package furrr (check our github for updates on this!). In what follows, I will exemplify the use of specr using a simple simulated data set (taken from the documentation of specr).

For more information about the various functions , visit the package documentation.  You can find comprehensive examples and vignettes on how to use the various functions of the package on the following pages:

• Getting started: A comprehensive example. This vignette illustrates the major functions of the package.
• Customizing specification curve plots: This vignette exemplifies various ways to plot the specification curve.
• Decomposing the variance of the specification curve: An example of how to investigate variance components of the specification curve.
• Visualizing progress during estimation: This vignette explains how to create a customizable progress bar for longer computations.

A simple example

# install.packages("specr")

# install.packages("devtools")
devtools::install_github("masurp/specr")

# Load library
library(specr)

# The example data included in the package
head(example_data)
##         x1         x2         c1        c2        y1         y2 group1
## 1 1.533913  1.3697122  0.5424902 3.8924435 23.500543 10.4269278      0
## 2 1.680639  1.5163745 -1.2415868 3.3377268 17.017955  0.5733467      1
## 3 1.223941 -0.2381044  0.1405891 0.8911959 -3.678272  4.2303190      0
## 4 1.765276  0.9524049  4.0397943 1.8567454 21.668684 14.8865252      1
## 5 1.907134  0.6282816  3.1002518 5.5840574 32.713106 20.5251920      0
## 6 1.710695  1.1898467  0.4648824 4.0239483 20.422171  4.3471236      1
##   group2
## 1      A
## 2      C
## 3      B
## 4      B
## 5      A
## 6      C

# Run specification curve analysis
results <- run_specs(df = example_data, 
                     y = c("y1", "y2"), 
                     x = c("x1", "x2"), 
                     model = c("lm"), 
                     controls = c("c1", "c2"), 
                     subsets = list(group1 = unique(example_data$group1)))

# Check
results
## # A tibble: 48 x 12
##    x     y     model controls estimate std.error statistic  p.value
##    <chr> <chr> <chr> <chr>       <dbl>     <dbl>     <dbl>    <dbl>
##  1 x1    y1    lm    c1 + c2    4.95       0.525    9.43   3.11e-18
##  2 x2    y1    lm    c1 + c2    6.83       0.321   21.3    1.20e-57
##  3 x1    y2    lm    c1 + c2   -0.227      0.373   -0.607  5.44e- 1
##  4 x2    y2    lm    c1 + c2    0.985      0.324    3.04   2.62e- 3
##  5 x1    y1    lm    c1         5.53       0.794    6.97   2.95e-11
##  6 x2    y1    lm    c1         8.07       0.557   14.5    6.90e-35
##  7 x1    y2    lm    c1         0.0461     0.466    0.0989 9.21e- 1
##  8 x2    y2    lm    c1         1.61       0.394    4.10   5.72e- 5
##  9 x1    y1    lm    c2         5.15       0.625    8.24   9.95e-15
## 10 x2    y1    lm    c2         6.50       0.466   13.9    5.38e-33
## # … with 38 more rows, and 4 more variables: conf.low <dbl>,
## #   conf.high <dbl>, obs <int>, subsets <chr>

plot_decisiontree(results, legend = TRUE, label = T)

# basic summary of the entire specification curve
summarise_specs(results)
## # A tibble: 1 x 7
##   median   mad    min   max   q25   q75   obs
##    <dbl> <dbl>  <dbl> <dbl> <dbl> <dbl> <dbl>
## 1   3.55  4.73 -0.227  9.03 0.727  7.62   250

# summary by specific groups and  statistics
summarise_specs(results, 
                stats = lst(mean, median, min, max), 
                group = c("x", "y"))
## # A tibble: 4 x 7
## # Groups:   x [2]
##   x     y      mean median    min   max   obs
##   <chr> <chr> <dbl>  <dbl>  <dbl> <dbl> <dbl>
## 1 x1    y1    6.63   6.52   4.95  8.27    250
## 2 x1    y2    0.237  0.157 -0.227 0.727   250
## 3 x2    y1    7.80   7.84   6.50  9.03    250
## 4 x2    y2    1.36   1.34   0.653 2.14    250

# Plot specification curve analysis
plot_specs(results)

Conclusion

We do see a lot of value in investigating how analytical choices affect a statistical outcome of interest. However, we strongly caution against using specr as a tool to somehow arrive at a better estimate. Running a specification curve analysis does not make findings any more reliable, valid or generalizable than a single analyis. The method is only meant to inform about the effects of analytical choices on results, and not a better way to estimate a correlation or effect.

We hope that our package can help in this process. If you find any bugs or would like to propose changes, please do so via the respective Github page: https://github.com/masurp/specr/. We are looking forward to feedback and comments.

References

Conceptualizing online privacy literacy as a multi-dimensional knowledge concept

So far, online privacy literacy had mostly been operationalized as a pure knowledge concept. A few studies had shown that such a privacy knowledge is related to more self-data-protection. In a first step, we conducted an exhaustive qualitative content analysis of prior literature on privacy literacy and a profound content analysis of different sources (privacy policies, juridical documents, EU deliverables…) capturing a variety of aspects relevant to online privacy. The aim was to capture any aspect or area that we deemed related to online privacy literacy. What knowledge should people have? Which skills should they possess?

In the end, we came up with four knowledge dimensions: (1) knowledge about practices of organizations, institutions and online service providers (e.g., data collection, sharing, usage, and analysis); (2) knowledge about technical aspects of online privacy and data protection (e.g., understanding concepts such as cache, cookies, and how they relate to one’s privacy); (3) knowledge about laws and legal aspects of online data protection as well as (4) knowledge about European directives on privacy and data protection, and (5) knowledge about specific strategies for individual privacy regulation (e.g., what type of password to use, how to manage up privacy setting, etc). Based on these dimensions, we defined online privacy literacy as

a combination of factual or declarative (“knowing that”) and procedural (“knowing how”) knowledge about online privacy. In terms of declarative knowledge, online privacy literacy refers to the users’ knowledge about technical aspects of online data protection and about laws and directives as well as institutional practices. In terms of procedural knowledge, online privacy literacy refers to the users’ ability to apply strategies for individual privacy regulation and data protection (p. 339).

Based on the content analysis, we then developed a first item pool that we believed captured all aspects of online privacy literacy. We summarized the findings of the content analysis in the following publication:

• Trepte, S., Teutsch, D., Masur, P. K., Eicher, C., Fischer, M., Hennhöfer, A., Lind, F. (2015). Do people know about privacy and data protection strategies? Towards the “Online Privacy Literacy Scale” (OPLIS). In. S. Gutwirth, R. Leenes & P. de Hert (Eds.). Reforming European Data Protection Law.(pp. 333-365). Springer: Netherlands. https://doi.org/10.1007/978-94-017-9385-8 [Link | Preprint]

Development and validation of the online privacy literacy scale (OPLIS)

In a second step, our aim was to reduce the large item pool and develop a comprehensive, yet useable scale to measure online privacy literacy. Based on the initial item pool with 113 knowledge questions, we conducted three consecutive studies with which we tested the items for overall fit with the proposed theoretical concept. We measured all five dimensions identified in the qualitative content analysis. However, the analyses have revealed that the “legal” dimensions 3 and 4 were not distinct. The final scale hence includes only four dimensions of privacy literacy and consists of 20 items (five items per dimension).

Figure 1. The estimated bifactor model (Masur, Teutsch & Trepte, 2017).

The psychometric quality of the scale was tested using structural equation modeling. The results of the studies show that the multidimensional concept of online privacy literacy can be supported with empirical data. Yet, we found that the dimensions identified in the literature do not represent distinct knowledge areas. Instead, the common variance in all items accounted for a strong global factor that represented actual online privacy literacy. Accordingly, we suggested to model online privacy literacy as a bifactor model which accounts for both the global variance and domain specific variances.⁷

The model was validated with a quota sample of German internet users (N = 1.932). Based on different criteria the scale’s quality can be regarded as good. Specifically the global factor predicted different types of data protection behavior. The results of the three consecutive studies were published the the journal “Diagnostica”:

• Masur, P. K., Teutsch, D. & Trepte, S. (2017). Entwicklung und Validierung der Online-Privatheitskompetenzskala (OPLIS) [engl. Development and validation of the online privacy literacy scale]. Diagnostica, 63, 256-268. https://doi.org/10.1026/0012-1924/a000179 [Link | Preprint]

To make the scale available to the public, we also set up a website that features not only all items of the scale, but also instructions and tutorials on how to use them in different scenarios and norm tables for comparing individual results with the German population:

• www.oplis.de

A normative perspective on privacy literacy and “do it yourself”-data protection

Since its publication, the scale has been used in several studies – both nationally and internationally. Although higher literacy seemed promising in predicting people’s engagement in data and privacy protection, we started to question in how far fostering online privacy literacy was the right way to tackle the challenges of online privacy. In a chapter that was published as proceedings of the CPDP (a large conference on computers, privacy and data protrection), Tobias Matzner, Carsten Ochs, Thilo von Pape and I dove more deeply into the question of whether individual citizens should be responsible for their own data protection. Again, do-it-yourself (DIY) data protection has often been considered as an important part of comprehensive data protection. Fostering DIY privacy protection and providing the respective tools was seen both as important policy aim and as a developing market. Individuals are meant to be empowered in a world where an increasing amount of actors is interested in their data. The results of our scale development paper somewhat seemed to support this claim.

We took a step back and analyzed the preconditions of this view empirically and normatively: Thus, we asked first: Can individuals protect data efficiently? And second, should individuals be responsible for data protection. We found that individuals barely engage in sophisticated data protection that would allow them to restrict access of companies or institutions to their data and further argue that for normative reasons, a wider social perspective on data protection is required. Failing to see the larger, structural reasons behind individual lacks in privacy protection, does not attribute responsibility to the government as the actor who might actually be able to address structural problems. Seeing online privacy literacy as the solution thereby removes the responsibility of the government to ensure its citizens’ self-determination. The chapter has been published here:

• Matzner, T., Masur, P. K., Ochs, C. & von Pape, T. (2015). Self-Data-Protection – Empowerment or burden? In: S. Gutwirth, R. Leenes & P. de Hert (Eds.). Data Protection on the Move. (pp. 277-305). Springer: Netherlands. https://doi.org/10.1007/978-94-017-7376-8_11 [Preprint]

The role of online privacy literacy in democratic societies

In light of this, we started to question the role of online privacy literacy. By investigating the more recent literature on interventions and potential outcomes, we found that often enough, increasing people’s knowledge (even in all areas that were captured by OPLIS) did not necessarily result in a behavior change (e.g., less self-disclosure). It seemed even possible that people gained more knowledge through using certain privacy protection strategies and not vice-versa.

It was hence unclear whether a respective intervention actually contributes to more self-determination on the Internet and whether it makes people behave more deliberate and more in accordance with their privacy needs. Through continued discussions and reviews of the literature, we came to the conclusions that online privacy literacy should be conceptualized as a combination of knowledge and specific abilities and skills. For this purpose, a reconceptualization is proposed that includes privacy-related reflection and critical abilities as well as practical privacy and data protection skills next to factorial knowledge about economic, technical, and legal aspects of online privacy.

We argued that in order to be a true enabler for self-determination (including political responsibility), privacy literacy has to be viewed as a more complex set of skills, abilities and knowledge sets. The first installment of this model was published in the following paper:

• Masur, P. K., Teutsch, D., Dienlin, T. & Trepte, S. (2017). Online-Privatheitskompetenz und deren Bedeutung für demokratische Gesellschaften [engl. Online privacy literacy and its significance for democratic societies]. Forschungsjournal Soziale Bewegungen, 30 (2), 180-189. https://doi.org/10.1515/fjsb-2017-0039 [Link]

A reconceptualization of online privacy literacy

In my most recent publication, I developed this idea further and proposed a comprehensive model of online privacy literacy that outlines several factual privacy knowledge areas, privacy-related reflection abilities, privacy and data protection skills, and privacy-related critical thinking abilities (see Figure 2).

Figure 2: A comprehensive framework of online privacy literacy (Figure taken and translated from Masur, 2018).

With this model, I aim to provide a novel perspective on how to conceptualize privacy literacy. I agree that one of challenges today is that user are not literate enough to navigate online environments in a self-determined manner. But in contrast to prior literature, I do not think that is only a lack of knowledge that prevents them from becoming critical media users. Instead, I suggest that they (1) lack the knowledge to identify the risks involved in using online media, (2) do not have the necessary self-reflection abilities to questions and reflect their own behavior, (3) do not have the procedural skills to implemented increasingly more sophisticated privacy protection measures, and (4) lack the critical thinking ability to criticize the status quo and realize their own potential in actively changing their social and structural environments.

I argue that only by acknowledging reflection and critical thinking abilities, online privacy literacy may enable true self-determination. Particularly, individuals need to become critical in that they develop the ability to critically question and criticize institutional practices, the lack of usable data protection tools, and thereupon decide to change this status quote (e.g., by causing societal change through a certain voting behavior).⁸

With this paper, I am drawing from classical media literacy theory to arrive at a more comprehensive definition of what a literate Internet users and particularly a privacy literate user may look like. The theoretical paper was published in the German journal “Medien & Kommunikationswissenschaft”:

• Masur, P. K. (2018). Mehr als Bewusstsein für Privatheitsrisiken: Eine Rekonzeptualisierung der Online-Privatheitskompetenz als Kombination aus Wissen, Fähig- und Fertigkeiten [engl. It is more than just privacy risk awareness. A reconceptualization of online privacy literacy as a composition of knowledge, abilities, and skills]. Medien & Kommunikationswissenschaft, 66(4), 446 – 465. https://doi.org/10.5771/1615-634X-2018-4-446 [Link]

Conclusion and future perspectives

It has been questioned before why we need online privacy literacy a distinct concept – particularly as it (and even more so my latest conceptualization) is so close to classical media literacy concept. In the paper, I answered the following (translated from German):

Despite overlaps with some dimensions of media and digital literacy, I argue that the independent consideration of online privacy literacy makes sense both from a theoretical and an empirical point of view. Firstly, the focus on privacy-related knowledge dimensions, abilities and skills allows to explicit address necessary prerequisites for informationally self-determined behavior in online environments. Such a concept of literacy makes it possible to investigate particularly those person characteristics that actually allow the individual to determine for himself how his personal information and data is collected and used by potential third parties. Secondly, the concept allows for a conceptual condensation that is absolutely necessary for empirical assessments and imperative for potential educational programs designed to prevent irrational handling of one’s own privacy in online environments. Only the clear focus on privacy allows for reasonable operationalizations that are directly related to privacy-related behavior.

With this in mind, I believe that investigating online privacy literacy as well as its antecedents and consequences is an important goal. At this moment, several international project use OPLIS to investigate online privacy literacy in different cultures and contexts. Given that privacy must often be understood form a situational ⁹ (or at least a contextual perspective¹⁰ such studies are of utmost importance to understand the circumstances under which Internet users make use of their knowledge and skills and under which they don’t.

I will continue this research program in the next years aiming at understanding the connection between media and privacy literacy (is it a distinct concept or rather a subdimensions of a modern concept of media literacy?) and develop approaches toward assessing the various dimensions that I outlined in the latest paper. It is my hope that this research inspires other privacy scholars but also provides some valuable insights for practitioners and policy makers.