Why 2 × 2 interactions are hard to measure reliably
An interactive guide to signal, measurement error and covariance in repeated measures experiments, and to what they mean for reliability and standardised effect sizes.
1. Signal and error add as squares
Take a task with a 60 ms average interference effect. People differ in their true effect: for one person it might be 30 ms, for another 90 ms. Suppose the standard deviation of these true effects across people is 30 ms (τ = 30).
We never see a person's true effect. We estimate it as the difference between two condition means, each averaged over m trials. If single trials vary with standard deviation s, that estimate carries measurement error with variance 2s²/m. With s = 200 ms and m = 50 trials per condition, the error has a standard deviation of 40 ms.
Because measurement error is unrelated to who has a large or small true effect, the two sources are independent. Independent sources behave like the two sides of a right angled triangle: the spread of observed effects across people is the hypotenuse, √(30² + 40²) = 50 ms. Standard deviations do not add (30 + 40 is not 50), but variances do: 900 + 1,600 = 2,500.
Figure 1. True spread, trial noise, trial count and effect size
- moment to moment fluctuations such as attentional lapses and motor variability;
- trends within a session, such as practice and fatigue;
- differences between stimuli or items, when each trial uses a different word, face or question;
- binomial sampling on accuracy measures, where even a steady 50% chance of success varies across items.
2. Why differences between conditions are noisy
Suppose two people do a task with two conditions. Ann averages 500 ms in Condition 1 and 550 ms in Condition 2. Ben is slower, at 700 ms and 740 ms. In the left panel of Figure 2 each dot is one person. How far right a dot sits is that person's true Condition 1 score, and how far up is their true Condition 2 score, both measured from the group average. Ben's dot is up and to the right of Ann's because he is slower in both conditions.
A line through this plot is a recipe for turning each person's two scores into one number. Slide every dot straight onto the line by the shortest route, and where it lands is that person's score. The angle of the line decides which score you get. A horizontal line gives Condition 1 alone. The rising diagonal gives overall speed, because you move along it when both scores go up together, and on it Ann and Ben land far apart. The falling diagonal gives the difference between conditions, because you move along it when one score goes up and the other goes down. Ann's difference is 50 ms and Ben's is 40 ms, so on that line they land almost on the same spot.
People who are slow in one condition are slow in the other, so the cloud is a long thin ellipse lying along the rising diagonal. A line along the ellipse spreads people far apart, and a line across it squashes them together. Measurement noise behaves differently. Each condition mean carries its own noise, unrelated to the noise in the other condition, so noise is a round blob, drawn as the red circle. It blurs each person's landing spot by the same amount whichever line you choose. That blur is small next to the spread in overall speed and large next to the spread in differences, which is why difference scores are much less reliable.
The right panel shows where everyone lands on three lines: the overall level, the difference, and a line you turn with the angle slider. Only 45° and 135° are scores people use. The angles between them show the true spread shrinking smoothly as the line turns while the noise stays the same.
Figure 2. A line through the plot is a score
3. Four conditions: the 2 × 2 interaction
A 2 × 2 design has four conditions and four scores of this kind: the overall level, two main effects and the interaction. Scaled in the same way (every coefficient ±½), each of them carries exactly the same noise variance, s²/m. If people's true scores were equally correlated across all four conditions, the main effects and the interaction would also have the same true variance. The interaction becomes the hardest score to measure when people differ less in their interaction than in their main effects. The second slider in Figure 3 sets that ratio.
Figure 3. Noise and true variance in a 2 × 2 design
4. More trials raise reliability and dz
Averaging more trials shrinks only the noise block, s²/m. This raises reliability, and it also raises the standardised effect size dz, because dz divides the mean effect by the observed spread, which includes noise. As the number of trials grows, dz approaches a ceiling set by the true spread alone. When people barely differ in their true interaction, that ceiling is high and dz depends heavily on how many trials a study ran. The same raw effect can then produce very different dz values in two studies.
Figure 4. Reliability and dz as the number of trials grows
© 2026 Shane Lindsay. Text and figures licensed under CC BY 4.0.