Abstract

This is an absolutely superb paper tackling a hugely important policy question. The authors develop a new econometric approach to aggregating high-dimensional factorial designs in RCTs in order to identify the most effective policies, and apply it to the question of increasing childhood immunization rates. I am thoroughly impressed by every aspect of this paper: the dataset used, the specific treatment arms used, the policy relevance, and the approach to identifying the best policy.

Summary Measures1

We asked evaluators to give some overall assessments, in addition to ratings across a range of criteria. See the evaluation summary “metrics” for a more detailed breakdown of this. See these ratings in the context of all Unjournal ratings, with some analysis, in our data presentation here.2

Overall assessment: We asked evaluators to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”

Journal rank tier, normative rating (0-5): “On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here)” Note: 0= lowest/none, 5= highest/best”.

See here for the full evaluator guidelines, including further explanation of the requested ratings.

Written report

This is an absolutely superb paper tackling a hugely important policy question. The authors develop a new econometric approach to aggregating high-dimensional factorial designs in RCTs in order to identify the most effective policies, and apply it to the question of increasing childhood immunization rates. I am thoroughly impressed by every aspect of this paper: the dataset used, the specific treatment arms used, the policy relevance, and the approach to identifying the best policy. The overall takeaways of the paper are not only of particular policy relevance but also provide key insight into the economics of reminders.

A key strength of this paper relative to other papers in immunization is the outcome dataset. Most papers use parents’ self-reports of vaccination status, whereas this paper critically creates a new administrative database to accurately record immunization activity in both treatment and control. This is particularly important when evaluating treatments such as reminders because when parents self-report vaccinations, this kind of treatment may impact reporting itself irrespective of latent treatment effects on actual immunization

Banerjee et al. (2010)[1] is one of the most important and influential papers in this literature, illustrating the efficacy of incentives in increasing immunization rates. Another component of this current paper which I found very compelling was that it examines the key policy questions that remained following the results of that initial paper regarding the level of incentives over the course of the immunization course.

Another major gap in the literature that this paper addresses is on the role of SMS campaigns in preventative health. There is wide variation in the treatment effects from these types of messages, and it has thus far been unclear as to the reason why: is it necessary to have a famous Nobel Laureate deliver the message via video, for instance, as was done by some of the authors during COVID? One natural explanation for the heterogeneity in treatment effects is that SMS messages exhibit the kind of complementarity that is at the core of this design.

This paper is also impressive in the scope of its applicability. Not only is Haryana a huge state with many unvaccinated children, but the interventions are easily applicable to other states in India, and in fact other low-income countries. With the near universality of mobile phones across the developing world, there is no particular aspect of this treatment that does not apply to other settings. What’s more, even if there is a concern, the overall econometric method here is certainly portable, so even if the specific method for selecting ambassadors, for instance, needs to be modified given constraints on surveying from a census, tweaks on any specific arm can be easily incorporated into a near-replication of this study.

My only real feedback is that I think the point that the authors make that immunization is so cost-effective, it is still probably the best use of money to fight childhood disease even if there are many payments to inframarginal households, should probably be emphasized more and earlier on (perhaps in the introduction), as that really shapes what the bottom line take-away is about the best policy from this paper. Relatedly, the question of whether there may be long-run persistence in these policies is in some ways not especially relevant: even if there is zero persistence year to year within a village (which for the record I doubt), it is still so cheap relative to the benefit of immunization that these policies are well worth it.

References

[1]Banerjee, A. V., Duflo, E., Glennerster, R., & Kothari, D. (2010). Improving immunisation coverage in rural India: clustered randomised controlled evaluation of immunisation campaigns with and without incentives. BMJ, 340(may17 1), c2220–c2220.

Evaluator details

1. How long have you been in this field?

   • I have been in this field about [range coded 7-10 years.]

2. How many proposals and papers have you evaluated?

   • I have reviewed about 30 articles for academic journals.