Doing effect size calculations for meta-analysis is a good way to lose your faith in humanity—or at least your faith in researchers’ abilities to do anything like sensible statistical inference. Try it, and you’re surely encounter head-scratchingly weird ways that authors have reported even simple analyses, like basic group comparisons. When you encounter this sort of thing, you have two paths: you can despair, curse, and/or throw things, or you can view the studies as curious little puzzles—brain-teasers, if you will—to keep you awake and prevent you from losing track of those notes you took during your stats courses, back when. Here’s one of those curious little puzzles, which I recently encountered in helping a colleague with a meta-analysis project.

A researcher conducts a randomized experiment, assigning participants to each of \(G\) groups. Each participant is assessed on a variable \(Y\) at pre-test and at post-test (we can assume there’s no attrition). In their study write-up, the researcher reports sample sizes for each group, means and standard deviations for each group at pre-test and at post-test, and *adjusted* means at post-test, where the adjustment is done using a basic analysis of covariance, controlling for pre-test scores only. The data layout looks like this:

Group | \(N\) | Pre-test \(M\) | Pre-test \(SD\) | Post-test \(M\) | Post-test \(SD\) | Adjusted post-test \(M\) |
---|---|---|---|---|---|---|

Group A | \(n_A\) | \(\bar{x}_{A}\) | \(s_{A0}\) | \(\bar{y}_{A}\) | \(s_{A1}\) | \(\tilde{y}_A\) |

Group B | \(n_B\) | \(\bar{x}_{B}\) | \(s_{B0}\) | \(\bar{y}_{B}\) | \(s_{B1}\) | \(\tilde{y}_B\) |

\(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) | \(\vdots\) |

Note that the write-up does *not* provide an estimate of the correlation between the pre-test and the post-test, nor does it report a standard deviation or standard error for the mean change-score between pre-test and post-test within each group. All we have are the summary statistics, plus the adjusted post-test scores. We can assume that the adjustment was done according to the basic ANCOVA model, assuming a common slope across groups as well as homoskedasticity and so on. The model is then \[
y_{ig} = \alpha_g + \beta x_{ig} + e_{ig},
\] for \(i = 1,...,n_g\) and \(g = 1,...,G\), where \(e_{ig}\) is an independent error term that is assumed to have constant variance across groups.

### For realz?

Here’s an example with real data, drawn from Table 2 of Murawski (2006):

Group | \(N\) | Pre-test \(M\) | Pre-test \(SD\) | Post-test \(M\) | Post-test \(SD\) | Adjusted post-test \(M\) |
---|---|---|---|---|---|---|

Group A | 25 | 37.48 | 4.64 | 37.96 | 4.35 | 37.84 |

Group B | 26 | 36.85 | 5.18 | 36.46 | 3.86 | 36.66 |

Group C | 16 | 37.88 | 3.88 | 37.38 | 4.76 | 36.98 |

That study reported this information for each of several outcomes, with separate analyses for each of two sub-groups (LD and NLD). The text also reports that they used a two-level hierarchical linear model for the ANCOVA adjustment. For simplicity, let’s just ignore the hierarchical linear model aspect and assume that it’s a straight, one-level ANCOVA.

### The puzzler

Calculate an estimate of the standardized mean difference between group \(B\) and group \(A\), along with the sampling variance of the SMD estimate, that adjusts for pre-test differences between groups. Candidates for numerator of the SMD include the adjusted mean difference, \(\tilde{y}_B - \tilde{y}_A\) or the difference-in-differences, \(\left(\bar{y}_B - \bar{x}_B\right) - \left(\bar{y}_A - \bar{x}_A\right)\). In either case, the tricky bit is finding the sampling variance of this quantity, which involves the pre-post correlation. For the denominator of the SMD, you use the post-test SD, either pooled across just groups \(A\) and \(B\) or pooled across all \(G\) groups, assuming a common population variance.

Have an idea for how to solve this? Post it in the comments or email it to me. Need the solution because you have a study like this in your meta-analysis? Contact me and I’ll share it with you directly. I’m being coy because I’m teaching meta-analysis next semester, and I feel like this would make a good extra credit problem…