Abstract

Provides important evidence on effectiveness of a low-tech intervention to stem learning losses from school closures (likely to increase in coming years) by scaling it up vertically and across diverse geographies.

Note that results only apply to numeracy skills and for temporary and abrupt school closures. Future work should see if it is effective for other skills and circumstances (displaced people, girls), and if contextualizing content/methods improves outcomes.

Summary Measures

We asked evaluators to give some overall assessments, in addition to ratings across a range of criteria. See the evaluation summary “metrics” for a more detailed breakdown of this. See these ratings in the context of all Unjournal ratings, with some analysis, in our data presentation here.1

Overall assessment (See footnote2)

Journal rank tier, normative rating (0-5): On a ‘scale of journals’, what ‘quality of journal’ should this be published in?3 Note: 0= lowest/none, 5= highest/best.

Claim identification and assessment

I. Identify the most important and impactful factual claim this research makes4

SMS messages containing educational content along with nudges to engage in educational activities combined with short-duration tutoring sessions over a phone call, the content of which was adjusted through regular assessment of the student’s learning levels, raised numeracy skills (measured as ‘levels’ and later standardised using values from the control group) by 0.30 to 0.35 standard deviations.

II. To what extent do you *believe* the claim you stated above?5

My subjective 90% interval for this is (0.10, 0.50) standard deviations and is primarily based on the results from table 4 of this paper which shows the country-wise effects of the interventions. An additional factor for this confidence interval is that this study addresses some of the concerns pointed out by Crawfurd et al., 2021 in its pilot, that of households self-selecting into the treatment.

Written report6

This is a review of a five-country study that has attempted to assess the impact of two phone-facilitated interventions to continue the education of students which was disrupted due to the Covid-19 pandemic. The interventions were 1) regular SMS messages containing educational content, as well as nudges to engage in educational activities, 2) the previous SMS intervention plus regular, short-duration tutoring over a phone call; with [the] content of the tutoring adjusted by frequent assessments so that teaching was targeted to the level of learning of the student. The control group had access to whatever material their respective governments provided, typically through television or radio broadcasts, or print-outs.

The primary purpose of the study was to scale-up a low-tech and multi-method intervention addressing the educational needs of children whose schooling gets disrupted due to crises. The study also evaluated the impact of the interventions on secondary outcomes such as student’s non-cognitive skills, student & caregiver perception of learning progress, caregiver willingness to pay for such an intervention, teachers’ (from the phone-call intervention) perceptions towards their profession, and potentially adverse economic outcomes on caregivers. The results were generally positive for the phone intervention – seemingly robust to several checks, though not so much for the SMS intervention. Secondary outcomes also appeared to show positive results.

Importance and strengths of this paper

The study contributes positively to research on education in emergencies (EiE), particularly for children whose access to schooling has been suddenly disrupted. The paper is important for the following reasons:

1. Disruptions to education can have negative outcomes due to the loss of learning caused by such a phenomenon. Loss of learning can lead to an incomplete education which can potentially lead to poor economic outcomes (Hanushek & Woessmann, 2020)[1] or worse, especially for marginalised sections of the society (World Bank Group, 2018).

2. At the time of writing this review in 2024 disruptors to education such as extreme weather events, armed conflict, and disease outbreaks appear to be escalating and/or probabilities of them appear to be increasing (Gowan & Davis, 2024[2]; Mia, 2023[3]; Mora et al., 2022[4]; Seneviratne et al., 2023[5]; Torkington, 2024)[6]. In light of these efforts to develop & test EiE interventions and the results of such efforts are valuable.

3. Besides education in emergencies, there is potential for this intervention to be employed elsewhere such as distance learning for individuals who are hard to reach or from low-income backgrounds. There is some indication for this (Nejezchleb, 2020)[7] though more empirical research is necessary.

What are the specific strengths or commendable aspects of this paper?

1. The strength of this paper is not in the novelty of the topic or the interventions (since it has been mentioned by the authors themselves that this was developed and tested in an earlier pilot study in Angrist et al., 2022)[8] but in what it does – refining, scaling up, and providing evidence on the effectiveness of relatively-more accessible EiE interventions.

2. The heterogeneous contexts across which this study takes place (of note an extreme weather event in one study location) and the individual results for each country is another positive aspect of this paper. Through these we can infer if an intervention is generally effective or effective only in a specific context.

3. The third strength, and this is a catch-all point for a lot of things, is making the maximum possible use of a large-scale study by investigating any bias via robustness checks as well as looking at the effects this intervention might have on a range of secondary outcomes from perception of progress to adverse economic outcomes on caregivers.

Identifying and assessing the paper’s primary claims

1. Claim #1 [SMS + Nudges + Tutoring —> Numeracy]:7 SMS messages containing educational content along with nudges to engage in educational activities combined with short-duration tutoring sessions over a phone call, the content of which was adjusted through regular assessment of the student’s learning levels, raised numeracy skills (measured as ‘levels’ and later standardised using values from the control group) by 0.30 to 0.35 standard deviations.

   i) The source for this claim is section I of the paper. The primary results from table 1 in the appendices report a range of figures between 0.32 and 0.40 standard deviations.

   ii) My subjective 90% interval for this is (0.10, 0.50) and is primarily based on the results from table 4 of this paper which shows the country-wise effects of the interventions. An additional factor for this confidence interval is that this study addresses some of the concerns pointed out by Crawfurd et al., 2021[9] in its pilot, that of households self-selecting into the treatment.

   The confidence for this claim8 would be improved by robustness checks that check all or a minimum of more than half the countries included in the study. At least three robustness checks are conducted in only one country each.

2. Claim #2 [SMS + Nudges —> Numeracy]: SMS messages containing educational content along with nudges to engage in educational activities alone raised numeracy skills by 0.08 standard deviations.

   i) The source for this claim is Section I and appendix Table 1 of the paper.

   ii) My subjective 90% interval for this is (-0.02, 0.18) and is entirely based on the results from Table 4 from this paper which shows the country-wise effects of the interventions.

3. Claim #3 [Minimal difference between government and NGO-hired teachers]: There is minimal difference in learning outcomes for the phone-call based intervention when tutoring (that’s part of the intervention) is carried out by government teachers versus when it is implemented by teachers hired by NGOs.

   ii) The probability of this claim being true would be placed at 70% on the basis of it being conducted across a large (N = 4941) and diverse sample, but still short of the sample drawn to estimate the results of the intervention’s direct effects. And also due to a lack of information on the exact analysis

   performed to determine equivalence of effects.

Identifying some of the paper’s secondary claims

Note [from evaluator]: This section does not include all of the secondary claims made in this paper. The claims identified (and not assessed) are selected based on the reviewer’s subjective opinions on what would be relevant for policy makers.

1. Claim #1 [Perseverance]9: Across countries a sample of students (N = 8962) was selected to examine the effects of this intervention on their non-cognitive skills, one of which was perseverance (measured as willingness to attempt a second riddle after solving one before). Students that were part of the phone-call intervention had a 2.9 percentage point higher probability of [being] willing to try a second riddle (compared to a baseline willingness of 83.3%).

2. Claim #2 [Unemployment]: In the Philippines and Uganda a sample of female caregivers (N = 2446) was selected to examine if the interventions contributed to unemployment. The SMS-only intervention appeared to

   reduce female caregiver unemployment by 2.9 percentage points while the phone-call intervention increased it by 1.9 percentage points (from a baseline unemployment of 42.9%).

3. Claim #3 [Caregivers’ perceptions]: Across countries a sample of caregivers (N = 9188) was selected to examine the effects of this intervention on their perception of their children’s progress. Caregivers that were part of the SMS-only intervention perceived on average a 0.07 standard deviation increase in their children’s numeracy levels while those from the phone-call intervention perceived on average a 0.11 standard deviation increase.

4. Claim #4 [Teacher morale]: In Nepal some of the tutors for the phone-call intervention were selected from a pool of government teachers. From a small sample of this pool (N = 290), those that were assigned to the phone-call intervention had a 15.8 percentage point higher probability of expressing a willingness to be a teacher again (compared to a baseline willingness of 73.5%).

Concerns

1. [Possible misinterpretation of takeup rates]10 In section III.D.5 it is mentioned that in Uganda there was low take-up of alternate learning mediums like radio (29%) or printed materials (22%) during Covid-19, however there is a possible misinterpretation as Uwezo Uganda, 2021 shows that use of any type of material is around 69%. Low take-up of any one type of medium may reflect individual preferences and not necessarily problems with access. Regardless, the fact that cellular phone access is more prevalent (as mentioned in the paper in section II.A) might still help reach out to those who continue to be left out.

2. [SMS intervention description] There appears to be little to no description or discussion of the SMS-only intervention given how it was still effective in certain contexts.

Limitations of the paper and potential ways it could be improved

1. [Robustness checks inconsistency]11 The battery of robustness checks in this paper although helpful can appear to be a bit misleading when taking into consideration that some are conducted on samples from individual countries (e.g. testing validity using back-checks, differences with assessment methods and randomising questions) while others are conducted across all countries included in the study (e.g. inclusion of country weights, fixed effects). Fewer but broad[er] robustness checks would be better over many and a mix of broad & narrow ones.

2. [Caregivers mediate access to mobiles] This intervention can also be impacted by how caregivers mediate access to mobiles, especially to girls (Damani et al., 2022)[10]. While the study appears to be balanced (based on tables A1 & A3), it is possible that there was lower participation in the study of female students from specific countries. But any inquiry on this would have perhaps overextended the scope of this query and as such it would be better to consider this as a point for future research.

3. [Disruption $\neq$ displacement] Based on the evidence presented this intervention would be best suited to situations where education has been suddenly disrupted and learning losses need to be prevented. It would be less suited to education of displaced people as their concerns can revolve around bureaucratic barriers to access education or education tailored to their background (Shuayb et al., 2023[11]; Shuyab, 2019)[12].

4. [Phone access may limit scaling] Although the intervention is a low-tech intervention, it is still a technological intervention and can face issues with accessibility of cellular phones/landline phones, charging, connectivity or cellular network rates (Houngbonon et al., 2021[13]; Khurana et al., 2022[14]; Tandon et al., 2024[15]; Valenza et al., 2022)[16]. Expanding this intervention to different contexts should ideally be preceded by scoping studies for its viability.

5. [Network shutdown issues] Cellular networks are also vulnerable to shutdowns by governments or extreme weather events (Richtel, 2011[17]; Saaliq, 2019[18]; Yancey-Bragg, 2024)[19], and though the study had a situation where a typhoon occurred and did not appear to significantly disrupt learning outcomes (table 7 of the paper), it is still noticeably lower than those not affected by it; with sustained outages potentially having an even a bigger impact. A possible work-around is to complement this intervention with conventional mediums like printed self-study material.

Future work12

1. Future work should focus on concepts beyond basic numeracy skills. Would this intervention work for more foundational concepts like number recognition or more advanced ones like calculus? Would this intervention work for other areas like social sciences, language, or social & emotional learning (SEL)?

2. Future work that wants to explore ‘novel’ avenues on this specific intervention can explore how adapting the intervention to local contexts in terms of educational content, tutoring methods and medium of instruction can possibly improve the effect of this intervention. Researchers can look at Raisch et al., 2024[20] an example of contextualisation of educational interventions. 

3. Another important area for future research – but one that would perhaps require substantially more investment in resources is if this intervention can go beyond just preventing learning losses and be used as a viable, alternative medium of learning i.e. for sustained learning, teaching entire semesters or courses over it.

Nit-picking

1. Error bars in figure 2 are faint.

2. Elaborating on the outcome/dependent variables in the Data or Empirical Strategy section instead of just in the footnotes of tables would have been preferred.

3. Elaborating why the learning variable is a ranked outcome would have been preferred.

Questions to the authors

1. Why not imitate Gneezy et al., 2019 in capturing & isolating the impact of ‘real-effort’ on the tests?

2. Why not use a real-effort question as a control variable instead of as an outcome variable in a robustness check?

3. Why use student grade as a control variable when baseline learning was already a control variable?

4. Why is India left out of Figure A4: Learning Curve – Improved Implementation and Targeted Instruction Across Trials?

5. Can a copy of the SMS instructions be provided?

References

[1]Hanushek, E., & Woessmann, L. (2020). The economic impacts of learning losses (Vol. 225) [OECD Education Working Papers]. https://doi.org/10.1787/21908d74-en

[2]Gowan, R., & Davis, I. (2024). Trends in armed conflicts. In SIPRI Yearbook 2024. Oxford University Press. https://www.sipri.org/yearbook/2024/02

[3]Mia, I. (2023). Armed Conflict Survey 2023: Editor’s Introduction. IISS. https://www.iiss.org/publications/armed-conflict-survey/2023/editors-introduction/

[4]Mora, C., McKenzie, T., Gaw, I. M., Dean, J. M., von Hammerstein, H., Knudson, T. A., Setter, R. O., Smith, C. Z., Webster, K. M., Patz, J. A., & Franklin, E. C. (2022). Over half of known human pathogenic diseases can be aggravated by climate change. Nature Climate Change, 12(9), 869–875. https://doi.org/10.1038/s41558-022-01426-1

[5]Intergovernmental Panel on Climate Change (IPCC). (2023). Climate Change 2021 – The Physical Science Basis. Cambridge University Press. https://doi.org/10.1017/9781009157896

[6]Torkington, S. (2024, August 5). Geopolitics replaces inflation as the top worry for central banks and sovereign wealth funds. World Economic Forum. https://www.weforum.org/agenda/2024/08/geopolitics-inflation-central-banks/

[7]Nejezchleb, A. (2020). Bridging the Digital Divide: Telephone Tutoring at the Center.

[8]Angrist, N., Bergman, P., & Matsheng, M. (2022). Experimental evidence on learning using low-tech when school is out. Nature Human Behaviour, 6(7), 941–950. https://doi.org/10.1038/s41562-022-01381-z

[9]Crawfurd, L., Evans, D. K., Hares, S., & Sandefur, J. (2021). Teaching and Testing by Phone in a Pandemic (591). Center for Global Development. https://docs.edtechhub.org/lib/H3J9AJ2Z

[10]Damani, K., Daltry, R., Jordan, K., Hills, L., & Evans, L. (2022). EdTech for Ugandan girls: Affordances of different technologies for girls’ secondary education during the Covid-19 pandemic. Development Policy Review, 40(S2), e12619. https://doi.org/10.1111/dpr.12619

[11]Shuayb, M., Hammoud, M., & Alsamhoury, O. (2023). Schooling Experiences and Outcomes of Refugee Children in Lebanon, Turkey, and Australia.

[12]Shuyab, M. (2019). A Critique of Education in Emergency and Humanitarian Contexts: Observations from the field.

[13]Houngbonon, G. V., Le Quentrec, E., & Rubrichi, S. (2021). Access to electricity and digital inclusion: evidence from mobile call detail records. Humanities and Social Sciences Communications, 8(1). https://doi.org/10.1057/s41599-021-00848-0

[14]Khurana, A., Levin, V., Luna-Bazaldua, D., & Lieberman, J. (2022). A Synthesis Report on Piloting of Remote Phone-Based Formative Assessment Solutions in Ghana, Nepal, and Pakistan. World Bank. https://doi.org/10.1596/37566

[15]Tandon, K., Mukherjee, H., & Mukherjee, H. (2024, June 28). India’s Jio, Bharti Airtel, Vodafone Idea hike call tariffs for first time in three years. Reuters. https://www.reuters.com/business/media-telecom/indias-bharti-airtel-hike-mobile-tariffs-july-3-2024-06-28/

[16]Valenza, M., Dreesen, T., & Kan, S. (2022). On Call: Using Mobile Technologies to Measure Learning in Emergencies. UNICEF Office of Research - Innocenti. https://eric.ed.gov/?id=ED624513

[17]Richtel, M. (2011). Egypt Cuts Off Most Internet and Cellphone Service—The New York Times. https://www.nytimes.com/2011/01/29/technology/internet/29cutoff.html

[18]Saaliq, S. (2019, December 21). India suspends internet and phone services to quell protests. AP News. https://apnews.com/article/643423717f6494718e449dac441c3db9

[19]Yancey-Bragg, N. (2024). “No excuse,” mayor says, for Helene victims to be isolated without cell service. USA TODAY. https://www.usatoday.com/story/news/nation/2024/09/30/hurricane-helene-cell-phone-outages/75449885007/

[20]Raisch, N., Bailey, R., & Jones, S. M. (2024). SEL Insights: Applying behavioral insights to social and emotional learning programs in global settings. Social and Emotional Learning: Research, Practice, and Policy, 4, 100056. https://doi.org/10.1016/j.sel.2024.100056

Akhter, N. (2022). Exploring the effectiveness of online teaching learning practices at the government primary schools of Bangladesh during the COVID-19 Pandemic [Thesis, Brac University]. https://dspace.bracu.ac.bd:8443/xmlui/handle/10361/17185

Are our Children Learning? Illuminating the Covid-19 Learning Losses and Gains in Uganda. Uwezo National Learning Assessment Report, 2021. (2021). Uwezo Uganda.

Barham, K. A. (2023). Learners’ Experiences with Emergency Remote Learning at the Palestinian University During COVID-19 in Light of the INEE Minimum Standards. In S. Affouneh, S. Salha, A. Tlili, & S. Abu-Eisheh (Eds.), Education in the Post-COVID-19 Era—Opportunities and Challenges (pp. 47–60). Springer Nature. https://doi.org/10.1007/978-981-99-7293-7_4

Fischer, E. M., Sippel, S., & Knutti, R. (2021). Increasing probability of record-shattering climate extremes. Nature Climate Change, 11(8), 689–695. https://doi.org/10.1038/s41558-021-01092-9

Not Educating Girls Costs Countries Trillions of Dollars, Says New World Bank Report. (2018, July 11). [Text/HTML]. World Bank. https://www.worldbank.org/en/news/press-release/2018/07/11/not-educating-girls-costs-countries-trillions-of-dollars-says-new-world-bank-report

Evaluator details

1. How long have you been in this field?

   • 1 year

2. How many proposals and papers have you evaluated?

   • None, have reviewed proposals for ethical review though - around 10 or 20 of them