Abstract

We organized two evaluations of the paper: "How Much Would Reducing Lead Exposure Improve Children’s Learning in the Developing World?". The evaluators were moderately positive, but offered critique of the non-systematic nature of this review and its departure from “best practices”. To read these evaluations, please see the links below.

Evaluations

1. Anonymous evaluator 1

2. Anonymous evaluator 2

Overall ratings

We asked evaluators to provide overall assessments as well as ratings for a range of specific criteria.

I. Overall assessment: We asked them to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”

II. Journal rank tier, normative rating (0-5): On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here.) Note: 0= lowest/none, 5= highest/best.

See “Metrics” below for a more detailed breakdown of the evaluators’ ratings across several categories. To see these in the context of all Unjournal ratings, with some analysis, see our data presentation here.1

See here for the current full evaluator guidelines, including further explanation of the requested ratings.2

Evaluation summaries

Anonymous evaluator 1

Strengths include excellent explanations of how the literature was harmonized for comparison and how various analytical methods were considered. Authors make compelling cases for both. The major limitations are around the actual systematic review methodology that is employed as well as the causal conclusions drawn. Authors have not used published best-practices in the conduct of a systematic review, and have not noted this as a limitation of the work. It will be critical to add methodological limitations. There is also a lot of causal language used that cannot be supported by the type of data presented (correlational), thus the suggestion is to focus on the associations or relationships between the IVs and DVs, rather than speaking about 'effects'.

Anonymous evaluator 2

This non-systematic review3 seeks to explore the relationship between lead exposure and children’s learning outcomes by updating existing meta-analyses. Its key strengths are 1) consideration of standardized test scores for reading and mathematics 2) use of a specification curve, and different ways to assess the impacts of publication bias. Critically, a quality assessment is missing and some control choices are unmotivated. This evidence captured in the review itself is likely not causal, though the authors do examine a broader literature on the cognitive implications of lead exposure that certainly goes beyond correlational associations. 

Metrics

Ratings

See here for details on the categories below, and the guidance given to evaluators.

Journal ranking tiers

See here for more details on these tiers.

Evaluation manager’s discussion

This synthesis project establishes the relationship between lead exposure and learning outcomes across a range of studies and contexts. Both evaluators found limitations in the approach used to identify the literature included in this analysis, the choice of outcome variables, and the use of specification curves. As such, we do not have high confidence in the point estimates calculated by the authors. Nonetheless, the policy simulations offered by the authors are novel, insightful, and interesting. Given that these are simulations and not statements of objective fact, they can be informative despite low confidence in the meta-analysis. However, they should not be interpreted as garneted likely effects, rather, they represent plausible outcomes given current information constraints. Future primary research should attempt to evaluate the effects of the considered policies on “real world” outcomes (such as education) as opposed to only measuring biochemical outcomes (such as BLL).

While the evaluators were moderately positive, they both offered critiques of the non-systematic nature of this review and its departure from “best practices”. We agree with the second evaluator that results from Google Scholar are inherently non-systematic because results almost always vary between machines given the algorithms Google Scholar uses, making it non-replicable. We were unable to replicate the search results reported ourselves, finding many more results than the authors reported. 

As we requested, the evaluators focused on areas of their own expertise. However, there are several aspects of the paper we would like to see evaluated more carefully in future. These include:

• the plausibility of the highlighted results (see ‘claim identification’) in the context of other work; overall quantification

• The particular adjustment methods used (e.g., for reporting bias’); other adjustments that might be warranted (e.g., for “type M error”)

• The use of random effects REML models

• Particular issues of causal inference/observational studies

  • considering confoundedness and adjusting/weighting for this; the ‘coefficient stability method’

  • IV and OLS estimates compared

• The policy simulation details and implications

Unjournal process notes

Why we chose this paper

This paper was selected due to the significant policy and equity implications. Authors make a compelling argument for large-scale action across countries. If the effects suggested by the authors are correct, the proposed lead reduction programs could meaningfully contribute to improved human capital production in LMICs, and a reduction in global disparities. We want to ensure that decision-makers understand how credible this evidence is as they use it to develop national policies.

How we chose the evaluators

Evaluators were selected based on their knowledge and expertise in the field. Evaluators had expertise in child development, education, and evidence synthesis. Evaluators were invited to submit reviews and compensated for their time.

Conflict of interests

Evaluators, the evaluation manager, and others involved in this review at the Unjournal have no conflicts to report.