Skip to main content

Evaluation summary and metrics: “Mental Health Therapy As a Core Strategy For Increasing Human Capital: Evidence From Ghana”

Summary, metrics and ratings, Manager's comments on Evaluation of "Mental Health Therapy ... Increasing Human Capital: Evidence From Ghana” by Barker et al (2021), paper published in AER Insights (2022) as "Cognitive Behavioral ..."

Published onApr 07, 2023
Evaluation summary and metrics: “Mental Health Therapy As a Core Strategy For Increasing Human Capital: Evidence From Ghana”
·

Preamble

Two evaluators considered the NBER working paper: “Mental Health Therapy as a Core Strategy for Increasing Human Capital: Evidence from Ghana” (Barker, Bryan, Karlan, Ofori-Atta, & Udry, 2021). They were asked to foll­­ow the general guidelines available here.  For this paper we did not give specific suggestions on ‘which aspects to evaluate’. During this process, the paper (or perhaps a very similar ‘forked’ version) was accepted for publication in the journal AER: Insights, with a changed title: “Cognitive Behavioral Therapy among Ghana's Rural Poor Is Effective Regardless of Baseline Mental Distress(Barker, Bryan, Karlan, Ofori-Atta, & Udry, 2022).  The second evaluator (and the first evaluator, to a lesser extent) became aware of the latter version in the process of writing their evaluation, and adjusted accordingly.

Note that both evaluators chose to remain anonymous, so we will not publish their names or identities.

The authors have been informed and given a chance to respond. We did not hear back from them within the two-week waiting period. The authors are still encouraged to respond, and if they do, we will post their response here.

Evaluation Manager’s Statement

The two evaluations were generally positive about this work and its important contribution to our understanding of the potential benefits of psychotherapy programs in developing country contexts. However, the evaluators did raise some concerns that would be useful to address. I’m enlisting them here:

  • Access to survey materials would be useful for future replication efforts; the authors have already made available the analysis code to facilitate this

  • Both evaluators highlight that there was no evidence of heterogeneity by baseline mental health; however, it would be helpful to clarify if this was due to analyses being underpowered to identify small heterogeneous effects.

  • Although sample attrition was not mentioned in the NBER working paper, it has been addressed in the AER published version; however, some of the appendix content (Appendix Table 3) can be clarified, especially where coefficients are 0. A table note or some such should suffice

  • One evaluator also offers useful suggestions regarding the outcomes of this research 

Metrics

In addition to written evaluations (similar to journal peer review), we ask evaluators to provide quantitative metrics on several aspects of each article. These are put together below.

Ratings

Eval 1

Eval. 2

Rating category

Rating (0-100)

Confidence (0-5)1

Rating (0-100)

90% CI (0-100)*

Comments (Eval. 2)

Overall assessment

75

4

75

(70, 84)

Advancing knowledge and practice

65

4

60

(55, 65)

Methods: Justification, reasonableness, validity, robustness

60

3

90

(82, 94)

Logic & communication

75

3

70

(62, 82)

I wish these categories were separated - would rank it high for logic but lower for communication. Even though this is an extremely well written (clear and easy to follow) paper, I struggled with some of the framing and messaging (i.e. higher-level communication).

Open, collaborative, replicable

50

3

90

(80, 95)

Data and code are provided alongside the published paper (https://www.openicpsr.org/openicpsr/project/164481/version/V1/view) I have not tried to reproduce any of the analyses as I do not have access to Stata, hence wider CIs. Readme file is detailed and seems clear enough.

Engaging with real-world, impact quantification; practice, realism, and relevance

75

4

50

(48, 52)

Relevance to global priorities

75

3

50

(40, 60)

While the overall topic is relevant to global priorities research, I am hesitant to rely on the results of this manuscript to make any direct policy recommendations given the short follow-up period, the selection of some of the measures, and the corresponding p-values

Predictions

Evaluator 1

Evaluator 2

Prediction metric

Rating (0-5)

Confidence (0-5)2

Rating (0-5)

90% CI (0-5)*3

Confidence (0-5)

What ‘quality journal’ do you expect this work will be published in?

Note: 0= lowest/none, 5= highest/best

NA - already published

N/A already published

On a ‘scale of journals’, what ‘quality of journal’ should this be published in?

Note: 0= lowest/none, 5= highest/best

4

4

4

4

Data presentation of metrics

Comments
0
comment
No comments here
Why not start the discussion?