← Research

Why 2 × 2 interactions are hard to measure reliably

An interactive guide to signal, measurement error and covariance in repeated measures experiments, and to what they mean for reliability and standardised effect sizes.

Shane Lindsay, University of Hull

1. Signal and error add as squares

Take a task with a 60 ms average interference effect. People differ in their true effect: for one person it might be 30 ms, for another 90 ms. Suppose the standard deviation of these true effects across people is 30 ms (τ = 30).

We never see a person's true effect. We estimate it as the difference between two condition means, each averaged over m trials. If single trials vary with standard deviation s, that estimate carries measurement error with variance 2s²/m. With s = 200 ms and m = 50 trials per condition, the error has a standard deviation of 40 ms.

Because measurement error is unrelated to who has a large or small true effect, the two sources are independent. Independent sources behave like the two sides of a right angled triangle: the spread of observed effects across people is the hypotenuse, √(30² + 40²) = 50 ms. Standard deviations do not add (30 + 40 is not 50), but variances do: 900 + 1,600 = 2,500.

Figure 1. True spread, trial noise, trial count and effect size

Error SD, √(2s²/m)
Observed SD
Reliability, τ² / observed variance
dz, mean effect / observed SD
Left: true spread and measurement error meet at a right angle, and the observed spread is the hypotenuse. Right: the squared areas add. Reliability is the grey share of the total square. Changing the mean effect moves dz and leaves reliability unchanged; changing the number of trials moves both.
What counts as measurement error here? It is anything that varies from one observation to the next when we estimate a person's condition mean from a finite number of trials or items:

2. Why differences between conditions are noisy

Suppose two people do a task with two conditions. Ann averages 500 ms in Condition 1 and 550 ms in Condition 2. Ben is slower, at 700 ms and 740 ms. In the left panel of Figure 2 each dot is one person. How far right a dot sits is that person's true Condition 1 score, and how far up is their true Condition 2 score, both measured from the group average. Ben's dot is up and to the right of Ann's because he is slower in both conditions.

A line through this plot is a recipe for turning each person's two scores into one number. Slide every dot straight onto the line by the shortest route, and where it lands is that person's score. The angle of the line decides which score you get. A horizontal line gives Condition 1 alone. The rising diagonal gives overall speed, because you move along it when both scores go up together, and on it Ann and Ben land far apart. The falling diagonal gives the difference between conditions, because you move along it when one score goes up and the other goes down. Ann's difference is 50 ms and Ben's is 40 ms, so on that line they land almost on the same spot.

People who are slow in one condition are slow in the other, so the cloud is a long thin ellipse lying along the rising diagonal. A line along the ellipse spreads people far apart, and a line across it squashes them together. Measurement noise behaves differently. Each condition mean carries its own noise, unrelated to the noise in the other condition, so noise is a round blob, drawn as the red circle. It blurs each person's landing spot by the same amount whichever line you choose. That blur is small next to the spread in overall speed and large next to the spread in differences, which is why difference scores are much less reliable.

The right panel shows where everyone lands on three lines: the overall level, the difference, and a line you turn with the angle slider. Only 45° and 135° are scores people use. The angles between them show the true spread shrinking smoothly as the line turns while the noise stays the same.

Figure 2. A line through the plot is a score

Left: true scores in two conditions, relative to the group average. The solid line is your line, with open circles where Ann and Ben land on it. Right: each mark is one person. The black bar spans one true SD either side of the average; the red bar spans one SD of measurement noise. Scores are distances along each line, which for the overall level and the difference means dividing the sum or difference by √2. This gives every line identical noise. Each condition has a true SD of 100 ms, and single trials have an SD of 200 ms.

3. Four conditions: the 2 × 2 interaction

A 2 × 2 design has four conditions and four scores of this kind: the overall level, two main effects and the interaction. Scaled in the same way (every coefficient ±½), each of them carries exactly the same noise variance, s²/m. If people's true scores were equally correlated across all four conditions, the main effects and the interaction would also have the same true variance. The interaction becomes the hardest score to measure when people differ less in their interaction than in their main effects. The second slider in Figure 3 sets that ratio.

Figure 3. Noise and true variance in a 2 × 2 design

Examples: ···
Noise per score, s²/m
Main effect reliability
Interaction reliability
Interaction dz
Each bar is one score's observed variance across people: the red block is noise, identical for every score, and the grey block is true variance. Each condition has a true SD of 100 ms, and single trials have an SD of 200 ms. The overall level bar runs off the scale.

4. More trials raise reliability and dz

Averaging more trials shrinks only the noise block, s²/m. This raises reliability, and it also raises the standardised effect size dz, because dz divides the mean effect by the observed spread, which includes noise. As the number of trials grows, dz approaches a ceiling set by the true spread alone. When people barely differ in their true interaction, that ceiling is high and dz depends heavily on how many trials a study ran. The same raw effect can then produce very different dz values in two studies.

Figure 4. Reliability and dz as the number of trials grows

Curves use the settings in Figure 3; change them there and these update. The dot marks the current number of trials. The dashed line in the right panel is the dz ceiling, the interaction effect divided by its true spread.
For experimentalists. Before correlating an interaction score with age, questionnaires or symptoms, compare its observed variance with the noise floor s²/m. Before comparing dz values across studies, check how many trials each study ran. Subtracting conditions removes what they share and keeps all of the noise.

© 2026 Shane Lindsay. Text and figures licensed under CC BY 4.0.