Abstract
1. I do not see any major technical issues with the paper.
2. The paper can be positioned more robustly i.e., the “need for it” needs to be better argued.
3. While the “effect” (mortality reduction) side of the paper has been done well, the “cost” part of the paper could be deepened. This would position it better and be supremely useful to policymakers.
Summary Measures
We asked evaluators to give some overall assessments, in addition to ratings across a range of criteria. See the evaluation summary “metrics” for a more detailed breakdown of this. See these ratings in the context of all Unjournal ratings, with some analysis, in our data presentation here.
| Rating | 90% Credible Interval |
Overall assessment | 85/100 | 75 - 95 |
Journal rank tier, normative rating | 3.7/5 | 3.2 - 4.2 |
Overall assessment: We asked evaluators to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”
Journal rank tier, normative rating (0-5): “On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here)” Note: 0= lowest/none, 5= highest/best”.
See here for the full evaluator guidelines, including further explanation of the requested ratings.
Written report
Summary
A truly needed and useful paper. I commend the authors on what they have produced.
The authors conduct a meta-analysis of mortality reduction from water chlorination. They use a clear procedure to discover and include studies into their analysis, restricting to randomized controlled trials for ease of inference. They use both frequentist and Bayesian models to derive odds-ratios (OR) for reductions in child mortality from a variety of water chlorination programs. The find a ~24% reduction in mortality from chlorination. This is a high-quality result with careful inclusion of studies and an extensive sensitivity analysis. With this estimate in hand, they conduct a cost-effectiveness analysis of three water chlorination interventions: chlorine dispensers in Western Kenya, in-line chlorination in India and an MCH based delivery for a general global context. They find that MCH delivery is the most cost-effective while dispensers and in-line chlorination are relatively more costly, though all three interventions are cost-effective compared to WHO benchmarks.
The summary of my comments is:
I do not see any major technical issues with the paper.
The paper can be positioned more robustly i.e., the “need for it” needs to be better argued.
While the “effect” (mortality reduction) side of the paper has been done well, the “cost” part of the paper could be deepened. This would position it better and be supremely useful to policymakers.
I have two major comments.
1. Study framing
My first comment has to do with positioning this study. Right now, it is positioned as serving the need for evidence on the cost effectiveness of water chlorination on reducing mortality. While I am personally very partial to this and think it is worthwhile, I suggest that the authors argue the premise more convincingly.
There is no doubt the quality of evidence in this paper is excellent. It is a rigorous, high-quality update since the last such meta-analysis. But it is one among many and I find it a little hard to believe that policymakers—both globally, and at the national and local levels in the health sector—are unconvinced that clean water is a top choice when allocating their funds. Are policymakers still genuinely skeptical about the cost-effectiveness of chlorination to reduce mortality, especially child mortality, in a low-cost manner?
That many policymakers do not actually allocate as much as they “ought to” is more the subject of a political economy analysis and not down to irrational choices on their part. Just because they are not implementing chlorination programs in a concerted, robust and enduring way does not mean they do not accept the premise – they are constrained in many other ways which may prevent this. Clean water has been part of global health policy focus for decades, everyone recognizes it as a high priority investment area. So rationalizing it—yet again—is likely not the issue holding back deployment of water treatment interventions.
Let’s accept that there is little to no need to convince anyone that water chlorination is absolutely the right thing to spend their money on to prevent illness and death. Given this, I would urge the authors to think a bit more about positioning this study.
One thought in this regard is that this study could be made more about which water chlorination intervention is best suited for a given context. In other words, going a bit beyond what this study is currently doing. To explicate this, right now we have three interventions in three unique contexts – dispensers in Western Kenya, in-line chlorination in India and an MCH based delivery for a general global context. What if I was a Kenyan health policymaker and saw this study:
I see that the MCH based chlorination is the most cost-effective. I think to myself maybe that is the first step I should take. But then I note that the estimate is for a generic global program, so now I am unsure. My MCH program is pretty robust and I know my costs probably fall on the lower end of the global distribution but I can’t be sure. Maybe I should just take the Kenya specific result for dispensers and be done with it?
I have just played out a hypothetical here. The thing that happened “well” is that a policymaker compared two possible ways to achieve a child mortality outcome through water chlorination – the primary purpose of a CEA like this – but what was weak in the above was that MCH does not have a truly local, Kenya-specific cost-effectiveness estimate that the policymaker could use, or no information that helps them get close to it. And this is true for an Indian policymaker, and other policymakers in lower-middle income and low-income countries. This relates to my second major comment below i.e., a way to better position this work
2. Deepen the cost modeling
My second major comment has to do with the “cost” part of cost-effectiveness: specifically, to deepen the cost analysis. I think it is important to add some useful sophistication to the cost model, while keeping it comprehensible. Being more thoughtful about cost modeling and analysis has begun to take hold among policymakers and donors. Deepening the cost part of this analysis offers one way for the authors to better position this paper.
The first layer of sophistication that can be added is to build out a cost model for each intervention which allows some variation. A full, high effort version is to:
Build a cost model that explicates the key cost elements e.g., materials, management, communications, transport and training, of a given intervention.
Then vary those elements – much like a sensitivity analysis – to get a variety of possible costs for the intervention.
Then, calculate a variety of cost-effectiveness estimates.
This enables a policymaker to “locate” their specific context in the cost distribution for a given water chlorination intervention and look up the relevant cost-effectiveness for it. A simpler way to generate a range of costs might be to develop a low, middle and high cost for dispensers, in-line and MCH then use those to derive a range of CEAs. Policymakers, readers and anyone else interested in this, will be able to locate their costs on the spectrum – maybe a policymaker has good information that—for dispensers let’s say—their context is close to the medium cost, therefore they will then use CEA estimates for the medium cost scenario. Sohn et al (2020) show how you can think a bit more deeply about modeling costs.
The second layer of sophistication is bounding/confidence intervals which might allow a decision maker to see beyond just the “mean case” and understand the extremes. For instance, in-line treatment might have lower mean cost effectiveness but the “worst case” (upper bound on cost per unit effect) for dispensers might be lower than that for in-line which may drive a risk averse decision maker to pick dispensers for their context. I do not have any great examples for you but you could think along the lines of what Wakker and Klaasen (1995) suggest.
General
Focus on CEA: I appreciate that you want to acknowledge the broader benefits of these interventions i.e., your calculation of net benefits. I think this is kind of distracting to be honest. Focus on the CEA and make it about choosing the best water chlorination intervention variant that yields the lowest cost per reduction in child mortality.
Relatedly, in your main manuscript, section “Cost-effectiveness”, you do not state the cost per death averted (this was available in a previous version of the paper). I understand the switch to cost per DALY averted but the major thrust of your quantitative analysis is mortality. Are there strong reasons not to have that? Are comparisons not possible with cost per death averted? Is it possible to re-introduce this?
Timeframe for inclusion of studies: The publication dates of the studies used span a 23-year period, from 1998 to 2021; one can assume that actual implementation lagged publication. Page 17 “Search strategy and selection criteria” says you used search criteria from past meta-analyses and updated to also include studies “…from February 2016 to May 2020”. Looking at the search strings in Table S1, the search set titled “Limits”, in which the search window seems to be 2012 to 2016, except for Ovid which is 2012 to “current”. This is all a bit unclear.
Can the authors:
State in very clear terms what their full search window was?
State why they chose this window, i.e., why start at a specific date (oldest you allow) and why end at a specific date (latest you allow)?
The second of these questions also relates to the age of some of the studies included i.e., the oldest being over 23 years old. Is it alright to allow studies that are over two decades old in the analysis? Supplementary Materials>4. Cost-effectiveness analysis>Drivers of cost-effectiveness states that baseline mortality is a major driver of cost effectiveness. One would assume that over time, health outcomes are improving in LMICs and LICs i.e., bumping the baseline levels of mortality downward.
Timeframes of studies: The studies have large variation in terms of follow up i.e., when outcomes of interest were measured (see unnumbered table at the end of Supplementary Materials>1. Details of included studies and comparison with other RCTs). This is partly addressed by the authors in the sensitivity analysis by showing that studies with shorter timeframes are not driving results.
However, this does raise the conceptual concern about what is meant by a water chlorination program in terms of timeframe, especially as it relates to the cost effectiveness. Currently, a five-year timeframe for implementation of the three representative interventions is used. That sounds reasonable but is there any reason (a) that we have the same horizon for all three (I can imagine something like in-line chlorination programming lasting longer than MCH or dispensers but I can imagine it the other way too) and (b) that we limit to thinking about a five-year horizon i.e., it may be that there is variation in cost-effectiveness based on differing time horizons (two, three, five, 10 and 15)?
I think some language to explain why a five-year horizon was chosen would be good to have.
Study weights: For the weights provided in Table S4, we are not given a description for the weighting scheme used for the frequentist model but are for the Bayesian model (2. Meta-analysis models > Study weights in Bayesian model). Possibly a non-issue, but just pointing this out. Any particular reason the weighting scheme for the frequentist model was not described?
Specific
Fig. 1: Very much appreciate the clear articulation of search strategy.
Figure 2 (A) the OR for Luby et al. 2006 is very large (a previous version of the paper showed a smaller OR for this). Why is this? Is this something to be concerned about? In table S3, I see that Luby et al 2006 report 2 deaths in treatment compared to zero deaths in control – this must be driving the large OR but surely there has to be some sensible way to constrain this rather than let it run away to a large number. Again, this is outside my technical expertise, so I defer to the authors. Just noting that anyone who reads this will see the strangely high OR for the Luby et al 2006 study in Fig. 2 (A).
Sensitivity analysis: I appreciate very much the sensitivity analysis which reassures us on the quality of the results.
Footnote 7 in Supplementary Materials>4. Cost-effectiveness analysis> Drivers of cost-effectiveness is missing its text.
Table S7, footnote F, last line has a citation missing – there’s a note that says “….IPUMS [add citation]);…”.
There’s a jump from table S7 to Table S13 in the Supplementary Materials.
Evaluator details
How long have you been in this field?
How many proposals and papers have you evaluated?
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
