Summary, metrics and ratings, Manager's comments on Evaluation of "Mental Health Therapy ... Increasing Human Capital: Evidence From Ghana” by Barker et al (2021), paper published in AER Insights (2022) as "Cognitive Behavioral ..."
Two evaluators considered the NBER working paper: “Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana” (Barker, Bryan, Karlan, Ofori-Atta, & Udry, 2021). They were asked to follow the general guidelines available here. For this paper we did not give specific suggestions on ‘which aspects to evaluate’. During this process, the paper (or perhaps a very similar ‘forked’ version) was accepted for publication in the journal AER: Insights, with a changed title: “Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress” (Barker, Bryan, Karlan, Ofori-Atta, & Udry, 2022). The second evaluator (and the first evaluator, to a lesser extent) became aware of the latter version in the process of writing their evaluation, and adjusted accordingly.
Note that both evaluators chose to remain anonymous, so we will not publish their names or identities.
The authors have been informed and given a chance to respond. We did not hear back from them within the two-week waiting period. The authors are still encouraged to respond, and if they do, we will post their response here.
The two evaluations were generally positive about this work and its important contribution to our understanding of the potential benefits of psychotherapy programs in developing country contexts. However, the evaluators did raise some concerns that would be useful to address. I’m enlisting them here:
Access to survey materials would be useful for future replication efforts; the authors have already made available the analysis code to facilitate this
Both evaluators highlight that there was no evidence of heterogeneity by baseline mental health; however, it would be helpful to clarify if this was due to analyses being underpowered to identify small heterogeneous effects.
Although sample attrition was not mentioned in the NBER working paper, it has been addressed in the AER published version; however, some of the appendix content (Appendix Table 3) can be clarified, especially where coefficients are 0. A table note or some such should suffice
One evaluator also offers useful suggestions regarding the outcomes of this research
In addition to written evaluations (similar to journal peer review), we ask evaluators to provide quantitative metrics on several aspects of each article. These are put together below.
Eval 1 | Eval. 2 | ||||
Rating category | Rating (0-100) | Confidence (0-5)1 | Rating (0-100) | 90% CI (0-100)* | Comments (Eval. 2) |
Overall assessment | 75 | 4 | 75 | (70, 84) | |
Advancing knowledge and practice | 65 | 4 | 60 | (55, 65) | |
Methods: Justification, reasonableness, validity, robustness | 60 | 3 | 90 | (82, 94) | |
Logic & communication | 75 | 3 | 70 | (62, 82) | I wish these categories were separated - would rank it high for logic but lower for communication. Even though this is an extremely well written (clear and easy to follow) paper, I struggled with some of the framing and messaging (i.e. higher-level communication). |
Open, collaborative, replicable | 50 | 3 | 90 | (80, 95) | Data and code are provided alongside the published paper (https://www.openicpsr.org/openicpsr/project/164481/version/V1/view) I have not tried to reproduce any of the analyses as I do not have access to Stata, hence wider CIs. Readme file is detailed and seems clear enough. |
Engaging with real-world, impact quantification; practice, realism, and relevance | 75 | 4 | 50 | (48, 52) | |
Relevance to global priorities | 75 | 3 | 50 | (40, 60) | While the overall topic is relevant to global priorities research, I am hesitant to rely on the results of this manuscript to make any direct policy recommendations given the short follow-up period, the selection of some of the measures, and the corresponding p-values |
Predictions
Evaluator 1 | Evaluator 2 | |||||
Prediction metric | Rating (0-5) | Confidence (0-5)2 | Rating (0-5) | 90% CI (0-5)*3 | Confidence (0-5) | |
What ‘quality journal’ do you expect this work will be published in? Note: 0= lowest/none, 5= highest/best | NA - already published | N/A already published | ||||
On a ‘scale of journals’, what ‘quality of journal’ should this be published in? | 4 | 4 | 4 | 4 |
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
