Abstract
We prioritized the paper “Effects of Emigration on Rural Labor Markets”[1] because of its potential relevance to the decision to shut down the charity “No Lean Season”. Although we normally seek two or more strong evaluations, we struggled to find evaluators for this paper. We would have persisted further, but we were also advised that other work in this area seems more relevant (see discussion below). Thus we decided to release this package with only a single evaluation.
Evaluation
Frank Gyimah Sackey’s evaluation
Overall ratings
We asked evaluators to provide overall assessments as well as ratings for a range of specific criteria.
I. Overall assessment: We asked them to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”
II. Journal rank tier, normative rating (0-5): On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here.) Note: 0= lowest/none, 5= highest/best.
| Overall assessment (0-100) | Journal rank tier, normative rating (0-5) |
Frank Gyimah Sackey’s evaluation | 70 | 2.5 |
See “Metrics” below for a more detailed breakdown of the evaluators’ ratings across several categories. To see these ratings in the context of all Unjournal ratings, with some analysis, see our data presentation here.
See here for the current full evaluator guidelines, including further explanation of the requested ratings.
Evaluation summary
Frank Gyimah Sackey
The paper sought to examine the effects of emigration on rural labour markets, specifically, how outmigration transforms rural labour markets [using an experiment involving a transport subsidy intervention]. Using a sample of 5,792 potential seasonal migrants across 133 villages, the authors observed that transport subsidies increased beneficiaries’ income [and this] generated spillover effects. The research findings indicate that the transport subsidies increase beneficiaries’ income due to better employment opportunities and also generated some spillovers [These included] increasing individual take-up rate as a result of a higher density offers which induces those connected to the offered recipients to also migrate, the village emigration rate increases, increase in the male agriculture wage rate in the village increasing income earned in the village. […]There is a decrease in the price of non-tradeables such as prepared food and tea while net food prices increase. The paper contributes immensely to the research on migration and labour studies. It also provides practical insights on intervention and adds useful value to research on interventions and has policy implications. The immense contribution is attributed to policy implications of the effects of subsidies on intervention and how the impact positively on such interventions. The practical insight is demonstrated by how recipients receive such interventions when they are directly involved and whatever benefits such as the subsidies go to them directly to achieve the desired results.
Metrics
Ratings
See here for details on the categories below, and the guidance given to evaluators.
| Evaluator Frank Gyimah Sackey | | |
|---|
Rating category | Rating (0-100) | 90% CI (0-100)* | Comments |
|---|
Overall assessment | 70 | (50, 90) | |
|---|
Advancing knowledge and practice | 70 | (50, 90) | |
|---|
Methods: Justification, reasonableness, validity, robustness | 70 | (50, 90) | |
|---|
Logic & communication | 70 | (50, 90) | |
|---|
Open, collaborative, replicable | 70 | (50, 90) | |
|---|
Real-world relevance | 70 | (50, 70) | |
|---|
Relevance to global priorities | 70 | (50, 90) | |
|---|
Journal ranking tiers
See here for more details on these tiers.
| Evaluator Frank Gyimah Sackey | | |
|---|
Judgment | Ranking tier (0-5) | 90% CI | Comments |
|---|
On a ‘scale of journals’, what ‘quality of journal’ should this be published in? | 2.5 | (1.0, 3.0) | |
|---|
What ‘quality journal’ do you expect this work will be published in? | 2.5 | (1.0, 3.0) | |
|---|
See here for more details on these tiers. | We summarize these as: 0.0: Marginally respectable/Little to no value 1.0: OK/Somewhat valuable 2.0: Marginal B-journal/Decent field journal 3.0: Top B-journal/Strong field journal 4.0: Marginal A-Journal/Top field journal 5.0: A-journal/Top journal
|
Evaluation manager’s discussion
Evidence Action shut down the charity No Lean Season (the precise intervention in this paper) partly because of "Mixed evidence of impact" (see GiveWell's report here). But the decision was also partly due to particular problems (allegations of fraud and mismanagement by the implementing partner. Thus, we suspected there may still be a strong case that this could be a cost-effective intervention. The present paper reported substantial positive effects, and was heavily cited. Thus we believed it deserved more careful evaluation, and if there were substantial flaws, these should be publicly shared.
Although we normally seek two or more strong evaluations, we struggled to find evaluators who could provide informed constructive criticism and appraisal. We might have persisted further, however…
After further consultation with an applied researcher in this area, we were advised that the GiveWell’s decision was more about issues having to do with the failure of the program to scale up, rather than its earlier performance as considered in the paper evaluated here. Follow-on work thus seems more relevant for further evaluation, such as “Delegation Risk and Implementation at Scale: Evidence from a Migration Loan Program in Bangladesh” (Mitchell et al, 2023) [2].
Author engagement
On June 26 2024 the authors let us know they are working on a revised version of the present paper, and hope to respond to the current evaluation when the revised version is released. We may revisit this work at that point, with potential for further evaluation.
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
