Skip to main content

Evaluation 1 of "How Much Would Reducing Lead Exposure Improve Children’s Learning in the Developing World?"

Evaluation of "How Much Would Reducing Lead Exposure Improve Children’s Learning in the Developing World?" for The Unjournal.

Published onJul 05, 2024
Evaluation 1 of "How Much Would Reducing Lead Exposure Improve Children’s Learning in the Developing World?"
·
1 of 2
key-enterThis Pub is a Review of
How Much Would Reducing Lead Exposure Improve Children’s Learning in the Developing World?

Abstract

Strengths include excellent explanations of how the literature was harmonized for comparison and how various analytical methods were considered. Authors make compelling cases for both. The major limitations are around the actual systematic review methodology that is employed as well as the causal conclusions drawn. Authors have not used published best-practices in the conduct of a systematic review, and have not noted this as a limitation of the work. It will be critical to add methodological limitations. There is also a lot of causal language used that cannot be supported by the type of data presented (correlational), thus the suggestion is to focus on the associations or relationships between the IVs and DVs1, rather than speaking about 'effects'.

Summary Measures

We asked evaluators to give some overall assessments, in addition to ratings across a range of criteria. See the evaluation summary “metrics” for a more detailed breakdown of this. See these ratings in the context of all Unjournal ratings, with some analysis, in our data presentation here.2

Rating

90% Credible Interval

Overall assessment

70/100

55 - 74

Journal rank tier, normative rating

2.5/5

2.0 - 3.0

Overall assessment: We asked evaluators to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”

Journal rank tier, normative rating (0-5): “On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here)” Note: 0= lowest/none, 5= highest/best”.

See here for the full evaluator guidelines, including further explanation of the requested ratings.

Written report

This review aims to examine the relationship between blood lead levels and learning (including IQ as well as reading and mathematics). They find that reducing blood lead levels by one natural log unit increases cognitive outcomes by 0.12 standard deviations. Authors do a great job of explaining how data was harmonized to allow for comparison across studies, and in walking the reader through their analytical choices. They also do a nice job of situating their work in the context of other previous reviews. However, gold standard methodologies for conducting a systematic review are not utilized, and the potential biases that this introduces into the results are not acknowledged. There are also causal claims that cannot be supported by the type of data used in this case.

[Managers’ note: we recognize that the paper also considers instrumental-variables estimates, which have a case for a causal interpretation under a particular set of conditions.]

Major comments

Results are given as one natural log unit increase in blood lead is associated with X. It would be helpful to know, on average, what the differences are in average blood lead levels for children in LICs vis a vis MICs or HICs earlier in the intro (before you start getting into the results).

Phrases like ‘causal link’, ‘causal interpretation’ and ‘effect of lead exposure’ do not seem appropriate given the study designs. In other places, more appropriate phrasing like ‘association’ and ‘relationship’ are used. Phrasing should be tempered throughout the manuscript to reflect that this is not causal evidence. Despite arguing the causal link later in the manuscript, the data itself cannot make this claim, particularly when there are so many comorbidities to lead exposure that are also related to lower cognitive achievement (e.g., SES).

Authors need to acknowledge the search is not comprehensive. They state that they seek to identify “all studies measuring the correlation between blood levels and IQ or test scores”, but then report that only existing SRs and Google Scholar were searched.

Please provide details on whether there were exclusions based on publication year or language.

On page 7, authors discuss that they include 286 estimates from 47 unique studies. It would be useful to know the minimum and maximum number of estimates contributed by a single study.

In Figures 3 and 4, please add the number of estimates being averaged for each study. It would also be useful to include the actual weighted average effect of the RVE model when referring to it in the Figure note. Finally, in Figure 4 there is a test of group differences presented (whether blood levels had a different effect of reading versus math?). It is unclear why this test was included or why such a difference would be expected. There is also no discussion of the test of group differences in text.

There needs to be a robust paragraph about the limitations of the systematic review methodology employed here. Gold standard SR methodology requires duplicate independent screening at full text to reduce bias. It also requires that risk of bias assessments are conducted in duplicate for each included study, and that results are presented with reference to potential biases in the underlying literature. It should also acknowledge whether limitations were put on the publication year or language of the studies, and should acknowledge that the search was not comprehensive (e.g., no grey literature sources were mentioned).

Figure A3 should include the study citations. It is very surprising that Taylor (2017)[1] would weaken the effect since it is a non-significant study without a particularly high weight. It would great if the authors could confirm this is indeed the correct study.

A PRISMA flow chart should be included to illustrate each phase of the search process as well as the reasons for exclusion at full text.

It would be useful to have additional information in A1 and A2, e.g., sample size, age at lead testing, age at cognitive testing, what was the measure used for cognitive testing (for reading and math).

Minor comments

In the introduction, the second sentence of the first paragraph does not follow logically from the first. You state that 600 million children in L&MICs have elevated blood lead levels, and then you state that 3% of this comes from HIC (which are not even mentioned in the first sentence). Do you mean that 3% comes from middle-income countries?

I am not sure who the target audience is, but when sigma is first used in the text (on page one of the intro) it may be useful to explain that this is the symbol for standard deviation.

On p. 31, in the first full sentence, please add the unit of measurement for the 149 and 15 numbers.

[Note: Copy-editing suggestions were removed here; these are useful to the authors, but not to other readers.]

References

[1] Taylor, C. M., Kordas, K., Golding, J., & Emond, A. M. (2017). Effects of low-level prenatal lead exposure on child IQ at 4 and 8 years in a UK birth cohort study. NeuroToxicology, 62, 162–169.

Evaluator details

  1. How long have you been in this field?

    • [20-25] years in research and [about 15] years in systematic reviews and meta-analysis. [Range coded to preserve anonymity.]

  2. How many proposals and papers have you evaluated?

    • Approximately 50

Connections
1 of 1
Comments
0
comment
No comments here
Why not start the discussion?