Abstract

This is a high quality paper that extensively studies the effects of therapy. I make a few small suggestions for analyses to explore the economic effects and the perceived efficacy of therapy.

Summary Measures

We asked evaluators to give some overall assessments, in addition to ratings across a range of criteria. See the evaluation summary “metrics” for a more detailed breakdown of this. See these ratings in the context of all Unjournal ratings, with some analysis, in our data presentation here.1

Overall assessment: We asked evaluators to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”

Journal rank tier, normative rating (0-5): “On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here)” Note: 0= lowest/none, 5= highest/best”.

See here for the full evaluator guidelines, including further explanation of the requested ratings.

Written report

This paper studies the long-run effects of therapy on depression, using two randomized trials of psychotherapy in India. The main result is that therapy reduces the likelihood of depression by 11 percentage points, five years after treatment. The evidence for this claim is strong, coming from follow-up data on randomized controlled trials with no evidence of problems in balance or attrition; deviations from the pre-analysis plan are clearly explained. The paper investigates the mechanisms of the causal effect, studies economic effects, and elicits beliefs about the perceived efficacy of therapy. The focus on long-run effects is important for informing policy.

The authors find no effect of therapy on economic outcomes like employment and earnings. This is explained by the sample being mostly women combined with social norms in India preventing female labor market participation. This leaves open the possibility that therapy affects economic outcomes for men, which is not tested. From Table A.1, there are roughly 70 men in the HAP trial follow-up (N=391 and proportion female of 0.82), which could be a large enough sample to test for an interaction effect. The authors note that the economic outcomes are not corrected for multiple-hypotheses testing and should be viewed as exploratory, so further exploratory work on the economic effects for men seems appropriate.

The authors report that therapy increased perceived effectiveness of therapy in both trials, despite the THPP trial being ineffective. One explanation, proposed by the authors, is that patients are not able to distinguish correlation from causation. Depression was reduced in the treatment and control groups in THPP at the same rate, presumably due to a common cause: pregnancy and childbirth (the sample is pregnant women). Hence, because their depression improved while receiving therapy, treated patients mistakenly believe that therapy caused the reduction in depression.

Since the treatment group beliefs about efficacy are actually higher in THPP than HAP, we might think that experimenter demand effects are playing a role. This is consistent with THPP being a longer trial than HAP 1 (6-14 sessions over 7-12 months in THPP compared to 6-8 weekly sessions in HAP), and hence being more likely to leave participants with fond memories of the trial. However, the evidence is mixed, since demand effects predict higher perceived efficacy in the treatment group relative to the control group, which is true for beliefs about efficacy at 1 year and 5 years after treatment, but does not hold for 3 months after treatment (Figure 4).2

Given the importance of understanding how perceptions of therapy affect take-up, it seems worthwhile to dig deeper into these results. The authors test for heterogeneous treatment effects for the effect on depression (Table A.3), and could repeat this analysis for perceived efficacy.3 For example, how does the effect of therapy on perceived efficacy vary by baseline expected efficacy?

The authors claim that $7 per month of depression averted is “remarkably cost-effective”, but it is not clear how to place this number in context. For example, it would be useful to know the cost-effectiveness of other therapy interventions. Moreover, it would also be useful to compare therapy against a cash transfer benchmark. Similarly, the discussion of policy-relevance could be expanded. How generalizable are the effects of a basic therapy like behavioral activation (which encourages engaging in enjoyable activities)? Do the effects reported here depend on characteristics specific to India?

The authors collected expert forecasts using the Social Science Prediction Platform. As shown in Figure 8, the interquartile range of experts’ predictions contains the actual estimate in only 3 out of 6 outcomes. This is beyond the scope of this paper, but for knowing how seriously to take these forecasts, it would be helpful to see a track record or some form of validation.

Evaluator details

1. How long have you been in this field?

   • [4-8 years — range coded to protect anonymity]

2. How many proposals and papers have you evaluated?

   • 10