Abstract

We organized two evaluations of the paper: "Water Treatment and Child Mortality: A Meta-analysis and Cost-effectiveness Analysis"[1]. To read these evaluations, please see the links below. The first was a team effort by Hugh Sharma Waddington and Edoardo Masset, who recently published a related systematic review; this evaluation focuses mainly on the meta-analysis and the studies therein. The second (anonymous) evaluation focuses more on the cost-effectiveness analysis. Both evaluators agree that this paper tackles an important question and provides an advance on our prior understanding of the effects of water treatment on child mortality. Both also found areas where the paper could be improved or clarified.

Evaluations

1. Sharma Waddington and Masset

2. Anonymous Evaluator 2

Overall ratings

We asked evaluators to provide overall assessments as well as ratings for a range of specific criteria.

I. Overall assessment (see footnote)1

II. Journal rank tier, normative rating (0-5):2 On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here.) Note: 0= lowest/none, 5= highest/best.

See “Metrics” below for a more detailed breakdown of the evaluators’ ratings across several categories. To see these ratings in the context of all Unjournal ratings, with some analysis, see our data presentation here.3 See here for the current full evaluator guidelines, including further explanation of the requested ratings.4

Evaluation summaries

Sharma Waddington & Masset

We evaluated the conduct, analysis and interpretation of the meta-analysis findings. The paper concerns an important topic for the global burden of infectious disease, particularly in middle-income countries where most of the included studies were conducted. It presents useful information on childhood mortality following water treatment and protection interventions. However, we had major concerns about conduct and reporting, which did not follow systematic review standards of transparency, and we also had major concerns about the interpretation of the findings for policy and research.

Anonymous evaluator 2

1. I do not see any major technical issues with the paper.

2. The paper can be positioned more robustly i.e., the “need for it” needs to be better argued.

3. While the “effect” (mortality reduction) side of the paper has been done well, the “cost” part of the paper could be deepened. This would position it better and be supremely useful to policymakers.

Metrics

Ratings

See here for details on the categories below, and the guidance given to evaluators.

Journal ranking tiers

See here for more details on these tiers.

Evaluation manager’s discussion

The Unjournal solicited two evaluations of the paper “Water treatment and child mortality: a meta-analysis and cost-effectiveness analysis.” The paper was flagged to us as important by a staff member at a major philanthropic organization, and our screening indeed suggested it could be highly influential in moving money to water treatment.

The paper does two things. It first pools evidence from many studies and so allows the authors to have the statistical precision to examine the effect of water treatment on child mortality directly. This is an improvement over past work that attempted to back out such an effect indirectly, by looking at much more common indirect signals of mortality like diarrhea.

The second thing the paper does is use the meta-analytic results to do a cost-effectiveness exercise to see how cost-effective is water treatment at avoiding death (or increasing DALYs). The main takeaway is direct estimates of water treatment show much higher reductions in mortality than do indirect methods. In turn, this means that water treatment is estimated to be much more cost-effective at averting deaths than was previously thought.

I solicited two reviews of this paper. While all the evaluators looked at the entire paper, I asked one team of evaluators to focus more on the meta-analysis and studies therein and a second evaluator to focus more on the cost-effectiveness analysis. Both evaluators agree that the paper tackles an important question and provides an advance on our prior understanding of the effects of water treatment on child mortality. Both also found areas where the paper could be improved or clarified. Our first evaluators, who worked in a team, raised many questions and helpful criticisms. These include:

1. Questions about study inclusion and exclusion and transparency.

2. Issues of data quality and analysis transparency, such as what appear to be undisclosed deviations from the pre-registration plan and changes in the odds ratios of the underlying studies between drafts.

3. Questions for scale up and policy, such as the representativeness of the sample, questions around mechanisms, and concerns around the persistence of the behavioural change.

Our second evaluation focused more heavily on the CEA model and had two major suggestions. The first was around the framing of the study. The second focused on ways to make the CEA more useful for policymakers, including:

1. Disaggregating costs to allow individual policymakers to essentially make selections off of a menu to match the CEA to their situation more closely.

2. Sensitivity analysis of the CEA, making use of the estimates of uncertainty from the meta-analysis.

The authors have responded to both evaluations and updated the working paper in response. While not all of the changes to the paper have yet been implemented, my personal reading is that the evaluators noted a number of ways that the paper could be improved, it in turn was improved, and while the headline result remains largely the same, we have now learned some new things about:

1. The stability of the results of various changes in the data

2. Additional important points in the CEA that were not clear to me up front, such as the important role of compliance in affecting the results

3. Important limitations in most existing economic research that prevents meta-analyses from being conducted or greatly limits their usefulness. In particular I’m thinking of participant flow diagrams in studies.

Anyone with an interest in this topic should read the evaluations and response in full as my summary comments contain some of my own interpretation and both the evaluators and authors may disagree with parts. A major advantage of open review is that one can read the exchange directly and form their own opinion.

I extend my sincere thanks to the evaluators and the authors for what I view as a fulfilling and transparent cooperative effort to reach the best possible answer to an important question.

Unjournal process notes

Note on versions: This working paper had two public drafts. The evaluators were recruited after the first (NBER) draft was posted online but when we learned that the authors were close to publishing a second draft we froze the process and waited for the newer draft to be published. The evaluation is of the second (May 2024) draft, though reviewers may at times make points that compare across the drafts as they read both.

Why we chose this paper

This work is very relevant to charity recommendations on water interventions.  From our team notes “Kremer’s work on WASH inform[ed] Evidence Action’s $64 million grant”. GiveWell was one of the funders of this research and they indicated a strong interest in having it evaluated. Furthermore, at least one senior academic and another senior applied researcher outside our team specifically suggested this.

For a ~nontechnical discussion, see GiveWell’s blog “Research Strategy: Water” (updated July 2024).

How we chose (and tasked) the evaluators

We selected evaluators with WASH research experience and we drew them from both the public health and development economics communities. As noted above “one team of evaluators was asked to focus more on the meta-analysis and studies therein and a second evaluator to focus more on the cost-effectiveness analysis.” The ‘team evaluation’ mode seemed particularly promising, and something we may pursue again in future evaluations.