A dozen doubts about GiveWell’s numbers
Summary
We raise twelve critiques of GiveWell’s cost-effectiveness analyses. Ten apply to specific inputs for malaria prevention, cash transfers, deworming. Two are relevant for more than one intervention. Our calculations are available in this modified spreadsheet of GiveWell’s cost-effectiveness analysesWe realise that the most up to date version of the 2022 CEA is version 5but we use version 4. However, we checked and found no substantial differences between versions 4 and 5.. We provide a summary of these first, then set them out in greater detail. We encountered these as part of our work to replicate GiveWell’s analysis. They do not constitute an exhaustive review.
Malaria prevention
- GiveWell assumes that a large fraction (34%) of the total effects of anti-malaria bed nets come from the extra income children will later earn due to reduced malaria exposure. However, GiveWell provides little explanation or justification for these numbers, which are strikingly, indeed suspiciously, large: taken at face value, they imply that bednets are 4x more cost-effective at increasing income than cash transfers.
We were unsure what the income-increasing effect of bednet should be, so we investigated a number of factors, which we enumerate as the next six critiques.
- The evidence GiveWell uses to generate its income-improving figure for bednets is from contexts that seem very different from that of the charities it recommends. While GiveWell does discount this evidence it uses for ‘generalisability’, we suggest a larger discount would be consistent with GiveWell’s reasoning regarding deworming, and would reduce the total cost-effectiveness by at least 10%.
- We present possible mechanisms for malaria prevention to income benefits, but they do not explain most of the effect. It’s unclear how this would change the cost-effectiveness results.
- We demonstrate how GiveWell’s analysis does not sufficiently adjust for differences in context by looking at malaria prevalence. Malaria prevalence was higher in the context of the evidence than in AMF’s context. When we adjust for this, we expect this will lead to around a 17% decrease in AMF’s total cost-effectiveness, with low confidence.
- GiveWell has missed some evidence for the effect of malaria prevention on income. We expect this decreases the total cost-effectiveness of AMF by 14%, with moderate confidence.
- GiveWell may underestimate the differences in benefits between women and men. We expect with moderate confidence that incorporating more evidence would decrease the total cost-effectiveness of AMF by 16%.
- GiveWell assumes, without clear justification, that the income effects of malaria prevention will endure in just the same way as they suppose the effects from deworming do. We’ve previously called deworming’s benefits into question (McGuire et al., 2022), incorporating the adjustment proposed there would decrease the total cost-effectiveness by AMF by 3.4% to 10.2% (low confidence)
We’ve presented these changes in cost-effectiveness for changing only one variable at a time. If we implement all of them at once (excluding critiques 3 and 7 for which we are more uncertain how they’ll change the results), then the effects decline by 29% (see row 202 of “AMF total” tab)This is different from simply summing the reductions because some of the variables that are changed interact with each other (e.g., the adjustment factor and the income increase)..
Cash transfers
- GiveWell’s analysis of the effectiveness of cash transfers only uses one study to estimate GiveDirectly’s immediate increases to consumption (Haushofer & Shapiro, 2013) when there are many more available. Using more data we show that recipients of GiveDirectly cash transfers are actually 70% richer than assumed in GiveWell’s cost-effectiveness analysis. Hence, the relative increase in consumption from GiveDirectly cash transfers decreases from 0.35 to 0.22 log-units (i.e., a decrease of 27%, which leads to an overall 38% decrease in cost-effectiveness).
Deworming
- GiveWell’s analysis relies on one long-term study from Kenya (the PSDP; Miguel & Kremer, 2004). We found another long-term study, by Liu and Liu (2019), which was from China. We’re not sure how comparable these are but, if we naively average the income effects, the cost-effectiveness of deworming goes up by 22%. If we add some adjustments based on context (although we have not fully considered these), then the effects of deworming decrease by 22%.
- GiveWell subjective apply a large 87% ‘replicability’ discount to deworming to (in part) account for the fact it relies on only one long-term study. Given there are now two studies, that indicates GiveWell could reduce the replicability discount by some fraction, which would increase the cost-effectiveness of deworming. If, for example, the replicability discount decreased to 80% (a total guess), then the effects of deworming would increase by 21%.
Intervention non-specific concerns
GiveWell split their cost-effectiveness analyses at the country level. However, they fail to consider the granularity across two important factors at the country-level that might change the cost-effectiveness of the charities:.
- Granularity about the age at which recipients would have died from a cause and how this relates to GiveWell’s moral weights.
- Household sizes differ across countries which would affect the calculation of household spillovers.