Abstract
We evaluated the conduct, analysis and interpretation of the meta-analysis findings. The paper concerns an important topic for the global burden of infectious disease, particularly in middle-income countries where most of the included studies were conducted. It presents useful information on childhood mortality following water treatment and protection interventions. However, we had major concerns about conduct and reporting, which did not follow systematic review standards of transparency, and we also had major concerns about the interpretation of the findings for policy and research.
Summary Measures
We asked evaluators to give some overall assessments, in addition to ratings across a range of criteria. See the evaluation summary “metrics” for a more detailed breakdown of this. See these ratings in the context of all Unjournal ratings, with some analysis, in our data presentation here.
| Rating | 90% Credible Interval |
Overall assessment | 50/100 | 40 - 60 |
Journal rank tier, normative rating | 2.5/5 | 2.0 - 3.0 |
Overall assessment: We asked evaluators to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”
Journal rank tier, normative rating (0-5): “On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here)” Note: 0= lowest/none, 5= highest/best”.
See here for the full evaluator guidelines, including further explanation of the requested ratings.
Written report
Note: To aid the reader, we (the managers) offer paragraph summaries in italicized block quotes
Introduction
The evaluators support systematic reviews and meta-analyses like these
As authors of a systematic review published last year (Sharma Waddington, Masset, Bick and Cairncross, 2023)[1], which included many of the same studies as the review by Michael Kremer and colleagues (hereafter the “review authors” or “reviewers”), we have been able to undertake a thorough evaluation of the review. Policy decision making should be based on the results of systematic, critically appraised evidence rather than single studies (Waddington et al., 2012)[2], hence we support the approach the reviewers have taken to collect and synthesize the evidence systematically.
Highly relevant
Furthermore, the topic of the review is highly relevant for global health policy, perhaps even more so than the review indicates. For example, interventions that aim to provide populations access to improved water supplies in quantity and/or quality have been shown in systematic reviews to be strongly associated with reductions in diarrheal illness (Wolf et al., 2022[3]; Ross et al., 2023)[4], yet a systematic link to reported mortality had not been made until recently (Sharma Waddington et al., 2023)[1]. This is important since an estimated 90 percent of disability adjusted life years (DALYs) for diarrhea are due to mortality, mainly in childhood, the remaining 10 percent coming from episodes of illness across the whole population. The Global Burden of Disease (GBD) estimates for drinking water supplies (Wolf et al., 2023)[5] are currently based on systematic reviews of illness reported by children’s carers, under the bold assumption that morbidity due to causes like diarrhea is closely correlated with mortality. By collecting and reporting data on losses to follow-up due to mortality contained in participant flow diagrams in reports of randomised controlled trials (RCTs) published in health journals, and by obtaining unpublished data on mortality from authors working in development economics and health, the review provides direct estimates of mortality in childhood from water treatment and protection interventions, which can potentially be used in future GBD calculations.
Other strengths: CEA, sensitivity analysis, etc.
We also believe there is much to praise about the review’s methodological ambition to provide information for decision makers. For example, it includes cost-effectiveness analysis of various interventions, which is rarely done in meta-analyses. It employs prediction intervals to estimate the impact of a new intervention, and in this way it aims to account for heterogeneity between studies. Finally, it undertakes sensitivity analyses including estimating frequentist and Bayesian meta-analyses, together with an assessment of small study effects that finds no evidence for publication bias for mortality outcomes, which is a very rare finding in the literature on intervention effects.
Our major concerns about the paper relate to how the systematic review and meta-analysis have been conducted and reported, and how the analysis has been interpreted, as we discuss below.
Systematic review conduct
An important aspect of systematic reviewing is transparency in conduct and reporting, which helps to ensure the analysis can be replicated by others. The reviewers conducted systematic searches for published RCTs on water treatment and protection interventions, harvesting data on all-cause mortality that were reported in participant flow diagrams in some studies, and contacting authors of RCTs to obtain unpublished data on all-cause mortality in childhood that were not already in the public domain. Systematic searches might miss studies, particularly if the searches cut across academic disciplines; in this case, RCTs of diarrhea morbidity have been published by economists and epidemiologists. A previous version of the review omitted several trials that were included in the systematic review and meta-analysis of water, sanitation and hygiene (WASH) and mortality by us (Sharma Waddington et al., 2023)[1], several of which have since been included in the meta-analysis as indicated by the reviewers.
Why were some studies excluded?
However, several studies of apparently eligible interventions, which reported all-cause mortality in participant flow diagrams, remain excluded from the analysis. These include Ercumen et al. (2015)[6] which reports all-cause mortality from two trial arms (chlorine plus safe storage and safe storage alone), and Bowen et al. (2012)[7], a long-term follow-up of another household water treatment (HWT) study that was included (Luby et al., 2006)[8]; both studies, while underpowered, reported higher mortality rates in the household water treatment group than in the control.
Lack of PRISMA approach; deviations from pre-registration
In some respects, the review is transparent about what was done. Although a systematic review protocol was not, to our knowledge, registered with any of the usual repositories for such studies (e.g., Campbell, Cochrane, Prospero), a pre-analysis plan was submitted to the AEA registry in June 2020. Fig 1 provides information about the search process and Table S2 provides information about which studies were excluded from the analysis, together with the reason why, although not in the usual form that a systematic review would provide. Reputable journals require systematic reviews to present a Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) study search flow diagram, discussion of excluded studies that users might reasonably expect to be included, and a PRISMA checklist that indicates, for example, deviations from protocol. There appeared to be deviations from the AEA registry record, such as the original exclusion of “cases where the study population is considered to be non-representative (e.g. interventions targeting HIV+ populations)” (Tan and Kremer, 2020)[9]. The review included a study of water filters and safe storage by Rachel Peletz and colleagues that was conducted among immunocompromised households (Peletz et al., 2012)[10]. We also wondered why a RCT on household water chlorination in Kenya (Kremer et al., 2008)[11] was not included in the analysis or in Table S2; this study aimed to evaluate the final pathway in water-borne diarrhea disease transmission by addressing contamination between source and point of use, and is therefore potentially highly policy relevant.
Differences between drafts; inter-rater assessments
We understand the working paper we have been sent to review is the second draft of a paper that has been online since 2023. We observed that the odds ratio estimates and 95 percent confidence intervals (95%CIs) differed, in some cases considerably, between the two working paper drafts. For example, we observed absolute differences in odds ratios of 0.04 or more for half (9) of the estimates (Boisson et al., 2013 [12]; Chiller et al., 2006 [13]; Dupas et al., 2021 [14]; Haushofer et al., 2020 [15]; Luby et al., 2006 [8]; Reller et al., 2003 [16]; Semenza et al., 1998 [17]; Peletz et al., 2012 [10]; Kremer et al., 2011 [18]), of which six (in bold) had differences of 0.08 or more. As a benchmark, the pooled effect in frequentist random-effects analysis was 0.75, hence these differences represent around one-third or more of the pooled effect magnitude. It is not clear to what extent the differences mattered for the findings, since in some cases the odds ratios were smaller, while others were bigger. However, we note that in the first draft of the review, the frequentist meta-analysis pooled effect for the chlorination sub-group was not statistically significant, whereas in the version evaluated by us, the review was able to find a significant effect, albeit over a slightly larger sample size. Hence, we believe it would be useful for the review to report inter-rater assessments on effect size data extraction and/or to indicate how discrepancies in the calculations were resolved, particularly regarding differences in estimated odds ratios and 95% confidence intervals.
Risk-of-bias assessment for RCTs; reporting bias for mortality vs morbidity
A key component of systematic reviewing is to undertake a transparent critical appraisal of the included evidence using risk-of-bias assessments, to help the reader understand how trustworthy are the findings from the included studies. The review does report a risk-of-bias assessment, using a tool that was developed primarily to assess observational studies, but the assessments are not discussed in the text or supplement. The review indicates that all studies are RCTs but, as is well known, RCTs can be at ‘high risk of bias’ due to problems in design or conduct, an example being selection biases due to high (or highly differential) losses to follow-up (attrition) in treatment and control arms, or joiners in cluster-RCTs. Another key aspect of the risk of bias concerns the quality of outcomes data collected. The review mentions that mortality data are more accurate and less biased than reported illness, even in unblinded trials. This is surely correct. The review could discuss in more detail what are the potential biases in reporting mortality, and why these are minor in comparison to biases in reported diarrheal illness. If there is any evidence or research supporting this claim it would be useful to report it. For example, one reason why the evidence suggests that reported mortality is very likely to be an unbiased measure (e.g., Wood et al., 2008)[19] is that it is very unlikely that a child’s carers might misremember or misreport it, whereas they may well do for a common illness like diarrhea (Sharma Waddington et al., 2023)[1].
The risk-of-bias ratings reported in the supplementary materials range between 4 and 7 out of a total possible score of 11. We note that evidence suggests it is not appropriate to determine overall bias using quality scales (Jüni et al., 1999)[20]. Authors of critical appraisal tools have instead shown that it is possible to assess overall bias based on transparent decision criteria (e.g., Eldridge et al., 2016[21]; Sterne et al., 2016)[22]. The review should comment on the implications of the risk-of-bias assessment for the confidence in the findings of the evidence base.
Meta-analysis and pooling subgroups
Regarding the meta-analysis that was conducted, the review reports an overall pooled effect together with a sub-group effect for chlorination. However, the review could also have reported pooled effects for filtration, where there were three estimates. Perhaps the reviewers felt that the Peletz et al. (2012)[10] study, conducted among immunocompromised groups, was not representative of general contexts; but we note that, even if that study was excluded, meta-analytical pooling can be undertaken provided there is more than one independent effect size.
Heterogeneity; Measuring adherence
However, a key purpose of meta-analysis is not just to estimate a pooled effect, but also to explain heterogeneity in estimates across studies. The review conducted analysis of adherence and length of follow-up, among other factors, finding no strong association between mortality and adherence, and a negative association for follow-up length – that is, there was no significant effect of interventions on mortality in childhood for follow-ups beyond 52 weeks, as has also been found for diarrhea morbidity (Waddington et al., 2009)[23]. It is very difficult to measure adherence accurately since it is impossible to prevent populations from drinking other (unimproved) water sources, and because other disease transmission mechanisms may be more or less important in highly contaminated environments. For example, when sanitation is classified as unimproved, so most people are openly defecating or sharing toilet facilities with people from other households and/or using facilities that do not adequately remove excreta from the environment, the primary sources of pathogens are fecally contaminated fingers, fields, floors, flies, food and fomites, as well as fluids if drinking water becomes contaminated too (Wagner and Lanoix, 1958)[24]. Perhaps drinking water is particularly susceptible to contamination in such circumstances, so HWT might be effective if you can get people that openly defecate to practise consistent water treatment and protection through intensive promotion or inline drinking water provision. On a similar note, the review did not discuss the interaction of the interventions with baseline environmental characteristics. The sensitivity analysis considered the baseline prevalence of diarrhea, and the review observed that the meta-analysis was not sufficiently powered to conduct a disaggregated analysis. However, the review could have examined or discussed how the results might differ in different contexts in greater detail, since this has been a major concern in the literature.
Bayesian meta-analysis issues; strong differences from frequentist results; generalizability
It is increasingly common to use Bayesian meta-analysis, an approach first proposed by Paul Hunter (2009)[25] for HWT using empirical bias correction factors. The review states that the mean effect estimated by the meta-analysis is specific to the sample considered (p.14). This is only partly true. The review considered between-study heterogeneity at the review level, and aimed to predict study effectiveness beyond the sample considered, as measured by the Bayesian uncertainty analysis. However, in the Bayesian meta-analysis, the posterior estimates for individual studies differed from the frequentist model, sometimes considerably; for example, the estimate for Luby et al. (2006)[8] shifts from a whopping OR=23.88 (95%CI=0.08, 7240) to OR=0.74 (95%CI=0.37, 1.49). It would be useful for readers, who may be less familiar with Bayesian meta-analysis, if the review can explain why these differences are so large.
The review presents the numbers of deaths in treatment and control groups for all of the studies included in the meta-analysis in Table S3. This has required great effort on the part of the reviewers, and stands to be useful to researchers working on the effectiveness of interventions to reduce mortality in childhood for years to come.
Metadata on intervention contexts
However, it is standard practice in systematic reviews and meta-analyses on WASH topics to report transparently on the populations, interventions and the counterfactual water supply and sanitation conditions too (e.g., Fewtrell and Colford, 2004[26]; Arnold and Colford, 2007[27]; Waddington et al., 2009[22]; Wolf et al., 2022[3]; Sharma Waddington et al., 2023)[1]. For example, the reader wants to know the interventions evaluated, the circumstances in which the evaluations were conducted, the types of populations covered, such as whether any were from immunocompromised groups, and the degree of movement up the drinking water and hygiene ladders afforded by the intervention. This information should be readily accessible, very preferably in the main text.
Report standalone vs joint interventions
The front section of the review does not clarify whether the studies included were standalone water treatment and protection interventions or whether they were implemented alongside other interventions. It is common for WASH interventions to be implemented in conjunction with other treatments, which also may have independent or interactive effects on morbidity and mortality. For example, as noted in the supplementary materials, several included study arms were of multicomponent interventions where HWT was provided alongside cookstoves (Kirby et al., 2019)[28] or sanitation and hygiene (Humphrey et al., 2019)[29]. One chlorine trial incorporated food hygiene education (Semenza et al., 1998)[17]. Other studies had hygiene and sanitation co-intervention arms that were excluded (e.g. Luby et al., 2018; Null et al., 2018), although the review states they were incorporated in sensitivity analysis. This point changes the interpretation of the results, in some cases in important ways. For example, since the focus of the review is all-cause mortality, it should be clearer which of the studies combined HWT with software and hardware that can affect children’s exposure to enteric or respiratory infection such as washing with soap and water, food hygiene or indoor cook-stoves. The conclusions may need to be qualified by observing that the results should be interpreted as approximations of the effects of ‘water treatment and protection’, if ‘water treatment and protection’ interventions were implemented as multicomponent packages including other activities affecting morbidity and mortality in childhood.
Reporting random effects weights, etc.
We have an additional point about the meta-analysis as it was reported. It is standard practice to report the random effects weight of each study in the meta-analysis, as well as relative and absolute between-study heterogeneity (I-squared and Tau-squared) for all analyses conducted including sub-groups. Having a low value of heterogeneity helps the reader understand if the pooled effect is likely to be valid across the sample of studies included in the meta-analysis. Values of Tau-squared are reported at the overall review level, but weights and sub-group heterogeneity statistics can be reported transparently in forest plots.
Publication bias; Power calculation approaches
Regarding the publication bias analysis, which provides a rare example where small study effects were not measured, we believe this is because most of the studies were not designed to measure mortality as a primary or secondary outcome. As noted by the reviewers, there may still be publication bias present (for example, mortality data from 29 studies were not obtainable, as discussed below). However, we are less convinced by the approach used to assess the statistical power of the meta-analysis. The review added null results (the post-hoc simulation on page 4) to the observed results and checked whether the meta-analysis still found a statistically significant effect. We wondered if post-hoc power calculations would be a simpler approach to address the same question. Perhaps the review could calculate the minimum detectable effect size or power of the meta-analysis as a function of the number of studies and see whether it is sufficiently powered to detect an effect size in the presence of publication bias.
Cost-effectiveness analysis
Choice of overall sample estimates; uncertainty in cost-effectiveness
Although it was not a primary aspect of our review (as requested by Unjournal editors), we also had concerns about the cost-effectiveness analysis. Firstly, the analysis used the Bayesian meta-analysis estimate across the whole sample of studies, including filtration, spring protection and solar disinfection. However, since two of the cost-effectiveness estimates directly concerned chlorination, it would seem more appropriate to use the pooled meta-analytic effects for chlorination alone in those cost-effectiveness analyses. Secondly, the review does not provide uncertainty estimates for the cost-effectiveness estimates with respect to either the confidence intervals on intervention effectiveness or sensitivity analyses to different cost scenarios or other assumptions (e.g., adherence rates).
Limitations of the WHO GDP threshold for decisionmaking
The review also made frequent use of the WHO GDP threshold. We note that many commentators within and outside the World Health Organization (WHO) have expressed their scepticism about this threshold and its use in decision-making (see for example this document for a review of debates on the GDP threshold within and without WHO). The GDP threshold is still widely used today, and the review is not exceptional in this. However, since the threshold has been criticized in many ways, we suggest that the review reports the limitations of using the threshold for decision-making, and explain how the threshold should be interpreted for decision-making purposes in this particular context.
Implications for policy
Lack of (evidence of) representativeness
The review states that “the studies included in the meta-analysis are broadly representative of the settings in which policymakers might implement water treatment programs” (p.15). It is hard to believe that 18 studies could represent the contextual variability one would find within and across countries and contexts within countries, especially when one considers that 14 of the 18 included studies were conducted in middle-income countries. It would be useful to understand who are the policymakers that would find this sample representative. Site-selection biases operate, whereby research sites selected for trials are those where there is the greatest contamination of drinking water and diarrhoea disease burden (Sharma Waddington et al., 2023)[1]. Perhaps it should be accepted that the sample is not representative of contextual variability. But if it is representative, we suggest adding some supporting evidence.
Routes of transmission; pathogens
It is important to understand the different routes of infection transmission, and which particular diarrheagenic pathogens drinking water treatment and protection can address in typical disease circumstances, in order to understand the relevance and generalizability of the findings for policy. Endemic diarrhea in L&MIC contexts is understood to be caused by exposure to viruses (especially rotavirus), protozoa (especially cryptosporidium) and bacteria (especially E.Coli and Shigella) (Liu et al., 2017)[30]. However, water treatment may not adequately address faecal contamination if the treatment technology itself is not efficacious in combating disease (Arnold and Colford, 2007)[26]. An example would be filtration, which is efficacious against bacteria and larger protozoans, but less so against common viruses like rotavirus. It also requires safe storage for sustained efficacy as there is no residual protection after water has been filtered. Chlorination kills bacteria and most viruses, and has the advantage of providing residual protection. But, in usual doses, chlorine is much less effective against protozoans like cryptosporidium and Giardia, common causes of severe diarrhea in low-income contexts, especially, but not only, among immunocompromised groups such as those living with HIV (Abubakar et al., 2007)[31].
Sustainability (persistence); Hawthorne effects
In order to understand generalizability of the findings from a review of behaviour change interventions, one also needs to understand if desired behaviours are practised and sustained, such as whether sufficient protective agents are applied to treat drinking water or adequate personal hygiene practised at the point of use so that contaminated hands or utensils are not placed in drinking water storage containers. One aspect of this is to assess rates of adherence and sustainability, as done in the review. The review did not find a significant association between adherence and mortality, which is likely due to the different measures of adherence used in the literature and the problems in measuring adherence to drinking water technology more generally, as discussed above. The only consistent relationship that was observed appeared to be the limited effectiveness of HWT after 6-12 months of follow-up. Factors associated with dis-adoption include users disliking the odour and taste of chlorinated water.
Much of the evidence on water treatment has come from RCTs conducted at zero or negligible financial costs to participants, with frequent follow-up by outsiders and disruption of normal domestic routines (the ‘mzungu effect’) (Waddington et al., 2009)[22]. There is therefore a high potentiality in these studies for Hawthorne effects, where being observed leads to greater efforts to adhere to treatment protocols, favouring the treatment group in unblinded trials. This bias is especially likely to occur when follow-up and measurement occurs frequently, as it does in many evaluations of HWT interventions. For example, in analysis that includes many of the studies used in the review, Pickering et al. (2019: e1143)[32] reported that “virtually all the evidence that promotion of… point-of-use water treatment with chlorine or flocculant disinfectant reduce diarrhea come from studies that had daily to fortnightly contact between the behaviour change promoter and study participant”. Hence, one useful analysis that the review could perform would be to examine the association between odds ratios and frequencies of follow-up visits by investigators. When there are lots of visits, the findings of the studies are unreliable guides to the effectiveness of real-world programmes that do not have frequent follow-ups, yet require participants to undertake behavioural modifications where children’s carers must always treat household drinking water while also ensuring that children never consume water from unsafe sources.
Need better RCT reporting standards in development economics, esp. CONSORT participant flow diagrams
In our opinion, what the review clearly highlights is that current standards for reporting of RCTs, especially in development economics, are not fit for purpose. Reputable journals publishing field trials in health require that CONSORT participant flow diagrams are reported, which show numbers of individual participants by study arm from recruitment of clusters and individuals within clusters, through follow-ups, together with important reasons for attrition like death (Moher et al., 1998)[33]. Without this information it is difficult to assess important threats to validity in these studies, which might occur due to problems in design and conduct. It is not sufficient to publish data openly, as many economics journals require, in order to assess them. For example, a key aspect of the internal validity in cluster-RCTs is knowledge about when and how individual participants were recruited, so that total and differential selection bias into the study from joiners can be assessed. The same follows for selection bias out of the study (attrition), although this is more commonly evaluated. It can also be useful to know who dropped out of the study between enrolment and randomization stages to evaluate external validity.
A recent survey by Chirgwin et al. (2021)[34] of WASH impact evaluations in L&MICs found that only half of trials in health had reported a study participant level CONSORT diagram, whereas no RCTs of WASH in economics had done so. 3ie has published CONSORT standards for RCTs in economics (Bose, 2010)[35]. What the review demonstrates is that this lack of participant flow reporting is extremely costly. Had the participant flows been reported transparently, there would have been less need for the reviewers to contact RCT authors to obtain the attrition data on all-cause mortality in childhood. The reviewers themselves noted this process was “time-consuming… and led to the loss of some data that was once available but is no longer available” (p.16), since there were 29 studies whose authors responded that the mortality data had not been collected, or had been collected but were no longer available, or who did not respond at all.
Reasons for previous lack of analysis of mortality, solutions
The review suggests that the reason why there has been hitherto limited analysis of mortality is because multiple testing of hypotheses prevents researchers from analysing the impact of the interventions on mortality. We are not convinced about this since the lack of reporting of mortality data is more likely due to the use of small samples, the difficulty of collecting mortality data, and apparently the lack of familiarity with reporting mortality data. Hence requiring these data be analysed as part of pre-analysis plans is unlikely to address the problem sufficiently. We believe a more effective solution would provide incentives for authors of RCTs to report participant flow diagrams, as are done in other fields, including RCTs measuring the impacts of HWT on diarrhea published in health journals. RCTs are costly to undertake financially and often require substantial time engagement by participants, so there are strong ethical and, as shown in the review, practical reasons for authors to report participant flows, and for reputable journals and commissioners to require them to do so.
There are similar standards for reporting systematic reviews and meta-analyses, which we discussed above, relating to the publication of protocols, reporting of deviations from protocol and adherence to PRISMA conduct and reporting standards. A key purpose of a systematic review protocol is to help reviewers avoid making results-based choices (consciously or otherwise). This does not mean that deviations from protocol are not allowed, just that they are explained.
COI issues/statements
Finally, we believe the positionality of the reviewers is not reported satisfactorily. It would be useful to know, for example, if the included RCTs conducted by the reviewers were appraised by different authors. Furthermore, one of the reviewers is a Board member of Evidence Action, the campaigning NGO that provided data on which two of the cost-effectiveness scenario estimates are based, and another is a principal investigator of two studies that led to the organisation’s earliest campaigns (Drinking Water Chlorination and Deworm the World). We might expect these associations to be mentioned in reviewer declarations due to the potential for conflicts of interest. For example, UKRI states: “the existence of an actual, perceived or potential conflict of interest does not necessarily imply wrongdoing on anyone’s part. However, any private, personal or commercial interests which give rise to such a conflict of interest must be recognised, disclosed appropriately and either eliminated or properly managed. Reporting, recording and managing potential conflicts effectively… can help to generate public trust and confidence.”
References
[1]Sharma Waddington, H., Masset, E., Bick, S. and Cairncross, S. (2023) Impact on childhood mortality of interventions to improve drinking water, sanitation, and hygiene (WASH) to households: Systematic review and meta-analysis. PLoS Med 20(4): e1004215. https://doi.org/10.1371/journal.pmed.1004215
[2]Waddington, H., White, H., Snilstveit, B., Hombrados, J.G., Vojtkova, M., Davies, P., Bhavsar, A., Eyers, J., Perez Koehlmoos, T., Petticrew, M., Valentine, J.C. and Tugwell, P. (2012). How to do a good systematic review of effects in international development: a toolkit. Journal of Development Effectiveness, 4 (3), 359-387. https://doi.org/10.1080/19439342.2012.711765
[3]Wolf, J., Hubbard, S., Brauer, M., Ambelu, A., Arnold, B. F., Bain, R., Bauza, V., Brown, J., Caruso, B. A., Clasen, T., Colford, J. M., Jr, Freeman, M. C., Gordon, B., Johnston, R. B., Mertens, A., Prüss-Ustün, A., Ross, I., Stanaway, J., Zhao, J. T., Cumming, O. & Boisson, S. (2022). Effectiveness of interventions to improve drinking water, sanitation, and handwashing with soap on risk of diarrhoeal disease in children in low-income and middle-income settings: a systematic review and meta-analysis. Lancet (London, England), 400(10345), 48–59. https://doi.org/10.1016/S0140-6736(22)00937-0
[4]Ross, I., Bick, S., Ayieko, P., Dreibelbis, R., Wolf, J., Freeman, M. C., ... & Cumming, O. (2023). Effectiveness of handwashing with soap for preventing acute respiratory infections in low-income and middle-income countries: a systematic review and meta-analysis. The Lancet, 401(10389), 1681-1690. https://doi.org/10.1016/S0140-6736(23)00021-1
[5]Wolf, J., Johnston, R. B., Ambelu, A., Arnold, B. F., Bain, R., Brauer, M., ... & Cumming, O. (2023). Burden of disease attributable to unsafe drinking water, sanitation, and hygiene in domestic settings: a global analysis for selected adverse health outcomes. The Lancet, 401(10393), 2060-2071.
[6]Ercumen, A., Naser, A.M., Unicomb, L., Arnold, B.F., Colford, J.M.C. Jr. and Luby, S.P. (2015). Effects of Source- versus Household Contamination of Tubewell Water on Child Diarrhea in Rural Bangladesh: A Randomized Controlled Trial. PLOS ONE. 2015 Mar 27;10(3):e0121907.
[7]Bowen, A., Agboatwalla, M., Luby, S., Tobery, T., Ayers, T., & Hoekstra, R. M. (2012). Association Between Intensive Handwashing Promotion and Child Development in Karachi, Pakistan. Archives of Pediatrics & Adolescent Medicine, 166(11), 1037. https://doi.org/10.1001/archpediatrics.2012.1181
[8]Luby, S. P., Agboatwalla, M., Painter, J., Altaf, A., Billhimer, W., Keswick, B., & Hoekstra, R. M. (2006). Combining drinking water treatment and hand washing for diarrhoea prevention, a cluster randomised controlled trial. Tropical Medicine & International Health, 11(4), 479–489. https://doi.org/10.1111/j.1365-3156.2006.01592.x
[9]Tan, B. and Kremer, M. (2020). Water Treatment and Child Survival: A Meta-Analysis. AEA RCT Registry No. AEARCTR-0005977. Available at: https://www.socialscienceregistry.org/trials/5977.
[10]Peletz, R., Simunyama, M., Sarenje, K., Baisley, K., Filteau, S., Kelly, P. and Clasen, T. (2012) Assessing water filtration and safe storage in households with young children of HIV-positive mothers: a randomized, controlled trial in Zambia. PLoS, ONE 7 (10), e46548. Doi:10.1371/journal.pone.0046548.
[11]Kremer, M., Null, C., Miguel, E. and Zwane, A.P. (2008) Trickle down: Diffusion of chlorine for drinking water treatment in Kenya. Available at: https://users.nber.org/~confer/2008/si2008/EEE/kremer.pdf.
[12]Boisson, S., Dangour, A. D., Watson, L., Cumming, O., Che, Y., Velleman, Y., … Uauy, R. (n.d.). Interventions to improve water quality and supply, sanitation and hygiene practices, and their effects on the nutritional status of children - Dangour, AD - 2013 | Cochrane Library. Retrieved from https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD009382.pub2/full
[13]Chiller, Tom M., Carlos E. Mendoza, M. Beatriz Lopez, Maricruz Alvarez, Robert M. Hoekstra, Bruce H. Keswick, and Stephen P. Luby. "Reducing diarrhoea in Guatemalan children: randomized controlled trial of flocculant-disinfectant for drinking-water." Bulletin of the World Health Organization 84, no. 1 (2006): 28-35.
[14]Dupas, R., Causse, J., Jaffrezic, A., Aquilina, L., & Durand, P. (2021). Flowpath controls on high‐spatial‐resolution water‐chemistry profiles in headwater streams. Hydrological Processes, 35(6). https://doi.org/10.1002/hyp.14247
[15]Haushofer, J., Kremer, M., Maertens, R., & Tan, B. J. (2021). Water Treatment and Child Mortality: Evidence from Kenya. National Bureau of Economic Research. https://doi.org/10.3386/w29447
[16]Reller, Megan E., Carlos E. Mendoza, M. Beatriz Lopez, Maricruz Alvarez, Robert M. Hoekstra, Christy A. Olson, Kathleen G. Baier, Bruce H. Keswick, and Stephen P. Luby. "A randomized controlled trial of household-based flocculant-disinfectant drinking water treatment for diarrhea prevention in rural Guatemala." American Journal of Tropical Medicine and Hygiene 69, no. 4 (2003): 411-419.
[17]Semenza, Jan C., Leslie Roberts, A. Henderson, J. Bogan, and C. H. Rubin. "Water distribution system and diarrheal disease transmission: a case study in Uzbekistan." The American journal of tropical medicine and hygiene 59, no. 6 (1998): 941-946.
[18]Kremer, M., Leino, J., Miguel, E., & Zwane, A. P. (2011). Spring Cleaning: Rural Water Impacts, Valuation, and Property Rights Institutions*. The Quarterly Journal of Economics, 126(1), 145–205. https://doi.org/10.1093/qje/qjq010
[19]Wood, L., Egger, M., Gluud, L.L., Schulz, K.F., Jüni, P., Altman, D.G., Gluud, C., Martin, R.M., Wood, A.J.G. and Sterne, J.A.C. (2008). Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. British Medical Journal, 2008; 336:601. Doi:10.1136/bmj.39465.451748.AD.
[20]Jüni, P., Witschi, A., Bloch, R. and Egger, M. (1999). The hazards of scoring the quality of clinical trials for meta-analysis. JAMA, 282: 1054-60. DOI: 10.1001/jama.282.11.1054
[21]Eldridge, S., Campbell, M., Campbell, M., Drahota, A., Giraudeau, B., Higgins, J., Reeves, B. and Siegfried, N. (2016). Revised Cochrane risk of bias tool for randomized trials (RoB 2) Additional considerations for cluster-randomized trials (RoB 2 CRT). Available at: https://sites.google.com/site/riskofbiastool/welcome/rob-2-0-tool/rob-2-for-cluster-randomized-trials.
[22]Sterne, J.A.C., Hernán, M., Reeves, B.C., Savović, J., Berkman, N.D., Viswanathan, M., Henry, D., Altman, D.G., Ansari, M.T., Boutron, I, Carpenter, J.R., Chan, A.-W., Churchill, R., Deeks, J.J., Hróbjartsson, A., Kirkham, J., Jüni, P., Loke, Y.K., Pigott, T.D., Ramsay, C.R., Regidor, D., Rothstein, H.R., Sandhu, L., Santaguida, P.L., Schünemann, H.J., Shea, B., Shrier, I., Tugwell, P., Turner, L., Valentine, J.C., Waddington, H., Waters, E., Wells, G.A., Whiting, P.F. and Higgins, J.P.T. (2016). ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. British Medical Journal, 2016;355:i4919. http://dx.doi.org/10.1136/bmj.i4919.
[23]Waddington, H., Snilstveit, B., White, H. and Fewtrell, L. (2009). Water, sanitation and hygiene interventions to combat childhood diarrhoea in developing countries. Systematic Review 01. 3ie, New Delhi. https://doi.org/10.23846/SR0017 Also published as: Waddington, H., & Snilstveit, B. (2009). Effectiveness and sustainability of water, sanitation, and hygiene interventions in combating diarrhoea. Journal of Development Effectiveness, 1(3), 295-335. https://doi.org/10.1080/19439340903141175
[24]Wagner, E.G., and Lanoix, J.N. (1958). Excreta disposal for rural areas and small communities. World Health Organization Monograph Series No. 39. WHO, Geneva. Available at: https://iris.who.int/handle/10665/41687
[25]Hunter, P.R. (2009). Household water treatment in developing countries: comparing different intervention types using meta-regression. Environmental Science and Technology, 43 (23), 8991-8997. https://doi.org/10.1021/es9028217
[26]Fewtrell, L. and Colford, J.M. (2004). Water, Sanitation and Hygiene: Interventions and Diarrhoea: A Systematic Review and Meta-analysis. World Bank, Washington, D.C. Available at: https://documents1.worldbank.org/curated/zh/916941468167378489/pdf/349600HNP0Water0Sanitation0Hygiene.pdf.
[27]Arnold, B.F. and Colford, J.M. Jr. (2007). Treating water with chlorine at point-of-use to improve water quality and reduce child diarrhoea in developing countries: a systematic review and meta-analysis. American Journal of Tropical Medicine and Hygiene, 76 (2), 354-364. PMID: 17297049.
[28]Kirby, M. A., Nagel, C. L., Rosa, G., Zambrano, L. D., Musafiri, S., Ngirabega, J. de D., … Clasen, T. (2019). Effects of a large-scale distribution of water filters and natural draft rocket-style cookstoves on diarrhea and acute respiratory infection: A cluster-randomized controlled trial in Western Province, Rwanda. PLOS Medicine, 16(6), e1002812.
[29]Humphrey, J. H., Prendergast, A. J., Gharpure, R., Mor, S., Viney, M., Dube, K., Lello, J., … Berendes, D. (2019). Putting the “A” into WaSH: a call for integrated management of water, animals, sanitation, and hygiene. The Lancet Planetary Health, 3(8), e336–e337. https://doi.org/10.1016/s2542-5196(19)30129-9
[30]Liu, J., Platts-Mills, J. A., Juma, J., Kabir, F., Nkeze, J., Okoi, C., Operario, D. J., Uddin, J., Ahmed, S., Alonso, P. L., Antonio, M., Becker, S. M., Blackwelder, W. C., Breiman, R. F., Faruque, A. S., Fields, B., Gratz, J., Haque, R., Hossain, A., Hossain, M. J.,... & Houpt, E. R. (2016). Use of quantitative molecular diagnostic methods to identify causes of diarrhoea in children: a reanalysis of the GEMS case-control study. Lancet (London, England), 388(10051), 1291–1301. https://doi.org/10.1016/S0140-6736(16)31529-X
[31]Abubakar, I., Aliyu, S.H., Arumugam, C., Usman, N.K. and Hunter, P.R. (2007). Treatment of cryptosporidiosis in immunocompromised individuals: systematic review and meta-analysis. British Journal of Clinical Pharmacology, 63 (4), 387-393. Doi: 10.1111/j.1365-2125.2007.02873.x.
[32]Pickering, A. J., Cherukumilli, & K., Ray, I. (2023). Evaluating the hidden costs of drinking water treatment technologies. Nature Water, 1(4), 319–327. https://doi.org/10.1038/s44221-023-00055-y
[33]Moher, D. (1998). CONSORT: an evolving tool to help improve the quality of reports of randomized controlled trials. Consolidated Standards of Reporting Trials. JAMA, 279, 1489-91.
[34]Chirgwin, H., Cairncross, S., Zehra, D., & Sharma Waddington, H. (2021). Interventions promoting uptake of water, sanitation and hygiene (WASH) technologies in low- and middle-income countries: An evidence and gap map of effectiveness studies. Campbell Systematic Reviews, 17, e21194. https://doi.org/10.1002/cl2.1194
[35]Bose, R. (2010). A checklist for the reporting of randomized control trials of social and economic policy interventions in developing countries: CEDE Version 1.0. Working paper 6. International Initiative for Impact Evaluation, New Delhi. Also published as Bose, R. (2010). CONSORT Extensions for Development Effectiveness: guidelines for the reporting of randomised control trials of social and economic policy interventions in developing countries. Journal of Development Effectiveness, 2 (1), 173-186. https://doi.org/10.1080/19439341003624441
Evaluator details
How long have you been in this field?
How many proposals and papers have you evaluated?
Since we have both worked as academics and grant funders for impact evaluations like RCTs, systematic reviews and meta-analyses for a combined total of approximately 50 years, we think we have evaluated over 250 each, so perhaps in excess of 500 in total.
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
