Summary measures

Overall assessment

Answer: 75

90% CI: (70, 84)

Quality scale rating

“On a ‘scale of journals’, what ‘quality of journal’ should this be published in?: Note: 0= lowest/none, 5= highest/best”

Answer: 4

Confidence: 4 out of 5

See here for a more detailed breakdown of the evaluators’ ratings and predictions.

Written report

Thank you for inviting me to review this manuscript. I was asked to review the version shared on NBER and I note for the purposes of this review I have read the NBER paper, as well as the now-published version in AER: Insights. In the published version, the authors have already addressed some of my original minor comments (e.g. clarity regarding report sample sizes per group and randomization process - now much clearer, e.g. through a flowchart). I find this manuscript to be very clearly written and sound in terms of experimental design.

Summary:

This study is an RCT delivered in Ghana, providing group cognitive behavioral therapy (CBT). Participants were reported to be from the general population, although notably the 40 compounds selected for inclusion were those reporting the lowest average household proxy means test score. Individuals were not targeted based on pre-existing mental health problems or levels of distress. Results are assessed after a 2-3 month follow-up. Somewhat surprisingly, there is no evidence of heterogeneity by baseline mental distress. Authors report improvements in a variety of outcomes of interest, broadly described as ‘mental and physical health, cognitive and socioemotional skills, and downstream economic outcomes’.

Rationale & Framing1

My main critical comment pertains to the rationale of providing CBT to ‘a general population of poor people’. I acknowledge that the included population does include a large proportion of individuals reporting distress (70% moderate or severe levels) and equally that the authors address the lack of specific targeting to a given extent. Nevertheless, I think this could have been further developed and the language made more precise. Communicating around the policy implications could be particularly important if such otherwise promising and effective interventions are to be scaled and adopted more widely.

There has been a real boom in investigations of therapy provision in low- and middle-income countries (LMICs) and I worry that the current work sits in the context of a growing literature that poses a risk of stigmatizing people who live in poverty and a danger of pushing towards the provision of psychosocial support as a primary vehicle to poverty alleviation, above and beyond much needed developmental work. This argument has already been posed by others such as Lant Pritchett (“Development work versus charity work”).2

Improvements in infrastructure and the availability of accessible mental health care globally are needed, as well as effective interventions that improve the lives of vulnerable people living in poverty. At the same time, mental health interventions focus on improving targetable individual-level factors rather than factors at a macro-level. In broader behavioral science contexts, others have potently argued that individual-level solutions have led ‘public policy astray’ and are a misuse of resources (Chater & Loewenstein 2022).

Follow-ups

A key step in further research on this topic should be longer follow-ups. This is a limitation not only to this particular work but also more broadly to psychotherapeutical interventions in LMICs (e.g. meta-analysis from Lund et al., 2022; NB working paper). For instance, as the authors acknowledge, in a related comparable paper with a longer follow-up, Haushofer et al. (2020) contrastingly find null effects unlike the positive effects reported here. The potential of time decay is a crucial limitation for me. More positively, in their working paper Lund and colleagues find some initial evidence to suggest that psychotherapy can somewhat outperform unconditional cash transfers both in terms of cost and effectiveness regarding health and economic outcomes. More research is needed to make stronger claims with greater confidence, but taking the present paper and the existing literature, providing psychosocial support to people in LMICs seems to be a promising way to improve lives and can effectively sit within a package of offered support to help empower and lift people out of poverty. Notably, as well, Lund et al. provide more exploratory evidence that pairing economic and therapeutic support outperforms therapy on its own.

Outcomes

● Authors describe that the intervention led to ‘meaningful’ average increases in the studied outcomes. I’m unsure if increases in reports regarding good mental health ‘0.53 days per month’ is particularly strong in this regard.

● As a clinical psychologist, the convention in my discipline is to consider p > .05 as non-significant so my disposition regarding some outcomes that are barely significant at the .1 level is one of skepticism (e.g. 0.53 days with better mental health, p = .097; on average 0.48 days when poor health prevented engaging in regular activities for people with distress, p = .097).

● While the improvements in digit span could likely be interpreted as based on increases in ‘bandwidth’ associated with CBT, other cognitive scientists have been more skeptical about the use of working memory tests in the first place and have suggested these tend to be stable and working memory largely not malleable; hence their use should occur in the context of a broader cognitive battery.

● Generally I find the created indices to be reported clearly, although a touch further detail on the exact procedure (weights?) would have been welcomed. I note in passing that there are some existing recommendations in my field that caution the use of composite measures as this can pose a challenge to clarifying the mechanisms of action. In this particular case I think the risk is fairly low, given that the mental health index includes exclusively self-report measures clustering around distress, and the socioemotional skills index is also self-report, clustering around self-control.

● The ‘downstream economic outcomes index’ also comprises entirely self-reported measures on ‘the number of work days missed due to poor mental or physical health, self-reported economic status… and a self-evaluation of expected economic status in five years.’ Including objective measures would provide a stronger degree of evidence. That an intervention targeting (in part) improvements in cognition and self-assessment changed self-reported measures pertaining to ‘self-evaluation’ is a sign that the intervention worked well in its primary domain, but further measurement specificity and validity will be beneficial in future work.

Mechanisms

● Hard to isolate effects of CBT vs socialization effects (therapy provided in groups of 10 people) but this is a general concern for group therapy interventions and a question for appropriate control condition selection.

● Although the authors propose one underlying mechanisms (CBT -> better cognitive and socioemotional skills -> improvements in human capital), other possible mechanisms are still likely (cf. Lund et al 2022) and future efforts should be directed to clarifying the causal mechanisms so interventions can be optimized.

Other

● Data for this trial were collected prior to the announcement of other interventions (including provisioning of a cash transfer). I appreciate the transparency regarding reporting this. I still wonder how likely it is that people who were surveyed extensively about their financial position did not anticipate any further trial. Hard to assess the likelihood of any further demand characteristics, though some have argued this is generally common for interventional contexts in LMICs.

● I note in passing that a broader cognitive battery with greater ecological validity in its tests would have provided further support that the scarcity/bandwidth framing is appropriate. Scarcity experiments have largely been lab-based and so more abstract with potential not to correspond to lived experience. I’m uncertain how helpful framing around ideas such as ‘poverty leads people to give into temptation’ really is. We know people managing restricted incomes may spend their finances in seemingly ‘suboptimal’ ways - for instance, choosing to buy sugar instead of more nutrition-dense foods but these choices may not reflect irrationality but a more diverse set of priorities (e.g. wanting to enjoy flavor after eating only rice).

● As another minor comment expressed with some uncertainty, I’m not sure about the validity/appropriateness of claims such as people living in poverty being constantly presented with ‘stimuli regarding their own status… an individual born into a poor farming family may misinterpreted his low income as evidence of his own low levels of talent…” and beliefs such as ‘my efforts never pay off” etc. These are certainly defensible positions (as is the whole theoretical framing of the paper) but I find it equally plausible that there might not be self-centering in attributing ‘blame’ for all. People can still experience distress while recognizing external factors (e.g. “my yield is bad because of the bad weather that affected my crops”). I then wonder if the external environment is not a target as well. Otherwise if we improve people’s mental health in the short-term via psychosocial intervention but do not protect against negative environment shocks by directly addressing macro-level/environmental factors, people can still be trapped in the vicious cycle between poverty and poor mental health (e.g. Lund et al 2011; Ridley et al. 2020).

Evaluator details

1. How long have you been in this field?

[about 3-5 years]… (in global mental health specifically)

2. How many proposals, papers, and projects have you evaluated/reviewed (for journals, grants, or other peer-review)?

[about] 20