Abstract

The paper studies an important question of [the] effectiveness of low cost education programs which could reduce learning losses faced by primary students due to shocks or emergencies in five developing countries.

Summary Measures

We asked evaluators to give some overall assessments, in addition to ratings across a range of criteria. See the evaluation summary “metrics” for a more detailed breakdown of this. See these ratings in the context of all Unjournal ratings, with some analysis, in our data presentation here.1

Overall assessment (See footnote2)

Journal rank tier, normative rating (0-5): On a ‘scale of journals’, what ‘quality of journal’ should this be published in?3 Note: 0= lowest/none, 5= highest/best.

Written report4

Comments from journal-tier ratings and “Overall assessment”

The paper studies an important question of effectiveness of low cost education programs which could reduce learning losses faced by primary students due to shocks or emergencies in 5 developing countries. …

… the importance of the question makes it one of the recent interesting topics in economics research. Additionally, since the low cost intervention has effectively been scaled across 5 countries, I think the study has made a unique contribution by contributing to the external validity of education in emergencies research.

The quality of the paper is very high. The contribution of the project, the credibility of the methods used to study causality, its impact on real world issues. Furthermore, the cost effectiveness nature of the intervention makes it very useful.

[Major] comments5

[…] The paper makes a unique contribution by scaling the RCT and providing medium effect sizes of the intervention. I present my concerns in the hope that it may improve the paper:

1. Missing health indicators6

   As the intervention was implemented in the aftermath of the COVID-19 pandemic, I am surprised to see that no information on child and caregiver health is provided. The econometric model does not include any health indicators in the baseline control variables.

2. More details on phone ownership

   I think the paper could benefit from [including] more details about how the phone tutorial worked, especially if the phone owner and the caregiver were two different individuals. Were there regular intervals during the phone calls due to unavailability of other phones or network problems? Did it have any impact on the learning gains? Did the teacher ensure that the caregiver did not change over time, especially since it was right after the pandemic? How did the teacher ensure that there was not more than 1 student benefiting per call? The phone was on speaker, right? So could it be possible that a student who already had the lesson joined the same lesson in another friend's house, essentially having a similar lesson (say addition) twice? This also feeds into the problem of inadequate robustness checks. A more detailed description of a typical phone tutorial will help.

3. Additional mechanisms

   As teachers and caregivers also seem to benefit from the intervention, it could be that improved teaching effectiveness and improved caregiver attention could be potential additional channels to explain the results.

Minor comments:

1) Section IIIB and IIIC can be shortened.

2) It could be worthwhile to scale up the randomisation of instruction delivery via NGOs or teachers to other countries than Nepal. Some countries may have social norms where government teachers are given more legitimacy than NGO workers (or vice versa) in regard to instruction delivery. In that case, you may find one-directional results in terms of effectiveness.

Note: 7

Other comments (from the ratings sections)8

Overall, I liked the simplicity of the RCT design and it was cleanly executed. The main results are presented in an interesting manner. My concerns are with the robustness checks and mechanisms.

The logical flow in the data analysis is consistent with the arguments set forth by the authors. There is some repetition of content in Sections III B and C (already covered in the Introduction) which can be avoided.

…With reference to discussion on methods and data analysis, the authors have done a good job. The tables and figures are labelled properly and explained well.

Evaluator details

1. How long have you been in this field?

   • I have finished my PhD between 2016-2021 and been an Assistant Professor of Economics since then.

2. How many proposals and papers have you evaluated?

   • Between 5-10