Abstract

We organized two evaluations of the paper: “Intergenerational Child Mortality Impacts of Deworming: Experimental Evidence from Two Decades of the Kenya Life Panel Survey”[1]. The authors find evidence for an intergenerational mortality benefit that could be large enough to make a difference to the debate over the effectiveness of deworming (relative to other impactful interventions like bednets). The evaluators generally find the paper credible, but raise some concerns (e.g., about consistency with the pre-analysis plan) and request some robustness checks. The authors responded briefly, noting they are revising the paper in response to these and other comments. To read these evaluations, please see the links below.

Evaluations

1. Anonymous Evaluation 1

2. Anonymous Evaluation 2

Overall ratings

We asked evaluators to provide overall assessments as well as ratings for a range of specific criteria.

I. Overall assessment (See footnote1)

II. Journal rank tier, normative rating (0-5): On a ‘scale of journals’, what ‘quality of journal’ should this be published in?2 Note: 0= lowest/none, 5= highest/best.

See “Metrics” below for a more detailed breakdown of the evaluators’ ratings across several categories. To see these ratings in the context of all Unjournal ratings, with some analysis, see our data presentation here.3

See here for the current full evaluator guidelines, including further explanation of the requested ratings.4

Evaluation summaries

Anonymous evaluator 1

A well-written and coherent paper. One of the first showing intergenerational effects of childhood health intervention in [a] LMIC, including cost-benefit analysis. Statistical power calculation not provided. Literature review is slightly one-sided. Some results in tables could be discussed further in text. Some typical outcomes [were] not present (nor mentioned), [… and ] if [they] exist in the data [it] would be interesting to see them.

Anonymous evaluator 2

This is a highly credible and important analysis of the intergenerational effects of a Kenyan deworming intervention. Such effects are important to document because they could meaningfully affect the cost-benefit analysis of such programs and also teach us about the mechanisms through which parents transmit advantage to their children. However, the analysis would be more credible if the authors were transparent about their deviations from the pre-analysis plan, which include the addition of (and exclusion of) outcomes and changes to the empirical specification. Also, more information on the “first stage”—the impact of randomization on receipt of deworming treatment—would be helpful.

Metrics

Ratings

See here for details on the categories below, and the guidance given to evaluators.

Journal ranking tiers

See here for more details on these tiers.

Unjournal process notes

Why we chose this paper

From the prioritization assessor

This paper has the potential to be highly influential and very important.

Some concerns were raised about potential lack of vetting or replication.

There is substantial debate over the effectiveness of deworming relative to other impactful interventions like bednets. Evidence for the effectiveness of deworming seems to be far more uncertain with much greater confidence intervals than for other interventions. Previous work, particularly that focusing on the long-term impact, seems to be inconsistent, or in some cases seemed face-value implausible. See fairly non-technical discussions:
GiveWell's updated estimate of deworming and decay — EA Forum

https://www.vox.com/future-perfect/2022/7/19/23268786/deworming-givewell-effective-altruism-michael-hobbes

And GiveWell’s recommendation change: Evidence Action's Deworm the World Initiative – August 2022 version

More evidence and more synthesis, and more careful scrutiny would seem to be high value; the magnitude of the estimates of the impact of deworming, and our confidence in these estimates seems pivotal to driving charitable funding and intervention recommendations.

The intergenerational mortality benefit seems substantial enough in magnitude to make a difference as confirmed by the paper authors, and by a very quick chat GPT-assisted back-of-the-envelope calculation and review. (See here, may contain errors, not checked.)  

Evaluation process

This evaluation process took longer than normal/desirable. We prioritized this for evaluation on Dec. 18, 2023. The initially assigned evaluation manager was overloaded, and we needed to change. We commissioned three evaluations, one of which did not meet our standards for posting (lack of detail, some misunderstandings). We also had some delays in finding relevant evaluators in this area.

The authors engaged with this process, sharing a brief response (linked below), and suggesting that they intend to respond further as a new version of the paper is released. We allowed the evaluators to adjust their content to this response. E2 modified their report very slightly, but neither evaluator chose to change their ratings.