Skip to main content

Evaluation Summary and Metrics: "Universal Basic Income: Short-Term Results from a Long-Term Experiment in Kenya"

Evaluation Summary and Metrics: "Universal Basic Income: Short-Term Results from a Long-Term Experiment in Kenya" for The Unjournal.

Published onJun 11, 2024
Evaluation Summary and Metrics: "Universal Basic Income: Short-Term Results from a Long-Term Experiment in Kenya"
·

Abstract

We organized two evaluations of the paper: “Universal Basic Income: Short-Term Results from a Long-Term Experiment in Kenya”[1]. To read these evaluations, please see the links below.

Evaluations

  1. Anonymous evaluation 1

2. Anonymous evaluation 2

Overall ratings

We asked evaluators to provide overall assessments as well as ratings for a range of specific criteria.

I. Overall assessment: We asked them to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”

II. Journal rank tier, normative rating (0-5):1 On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here.) Note: 0= lowest/none, 5= highest/best.

Overall assessment (0-100)

Journal rank tier, normative rating (0-5)

Anon. evaluator 1

93

4.7

Anon. evaluator 2

85

4.0

See “Metrics” below for a more detailed breakdown of the evaluators’ ratings across several categories. To see these ratings in the context of all Unjournal ratings, with some analysis, see our data presentation here.2

See here for the current full evaluator guidelines, including further explanation of the requested ratings.

Evaluation summaries

Anonymous evaluator 1

This is an evaluation of "Universal Basic Income: Short-Term Results from a Long-Term Experiment in Kenya".[2]

Anonymous evaluator 2

[Strengths]

  • Rigorous large-scale RCT design

  • Simple yet effective statistical approach

  • (Mostly) outstanding statistical analysis/interpretation & clear communication + Insightful analysis of cash-transfer effects in development context

[Limitations]

  • Limited relevance for generalized UBI in macro equilibrium

  • Potential biases in self-reported data, which is insufficiently discussed

  • Additional socio-economic context would enhance understanding

  • Some statistical results and interpretations from the paper remain unclear

Metrics

Ratings

See here for details on the categories below, and the guidance given to evaluators.

Evaluator 1

Anonymous

Evaluator 2

Anonymous

Rating category

Rating (0-100)

90% CI

(0-100)*

Comments

Rating (0-100)

90% CI

(0-100)*

Comments

Overall assessment3

83

(93, 100)

4

85

(65, 95)

5

Advancing knowledge and practice6

70

(93, 100)

7

85

(65, 95)

Methods: Justification, reasonableness, validity, robustness8

95

(90, 100)

9

85

(63, 95)

10

Logic & communication11

70

(80, 90)

12

76

(40, 91)

13

Open, collaborative, replicable14

90

(80, 100)

15

30

(10, 75)

16

Real-world relevance 17

90

(80, 100)

18

87

(74, 98)

19

Relevance to global priorities20

60

(40, 80)

21

70

(45, 90)

22

Journal ranking tiers

See here for more details on these tiers.

Evaluator 1

Anonymous

Evaluator 2

Anonymous

Judgment

Ranking tier (0-5)

90% CI

Comments

Ranking tier (0-5)

90% CI

On a ‘scale of journals’, what ‘quality of journal’ should this be published in?

4.7

(4.0, 5.0)

23

4.0

(2.5, 5.0)

What ‘quality journal’ do you expect this work will be published in?

4.7

(4.0, 5.0)

24

4.0

(2.5, 5.0)

See here for more details on these tiers.

We summarize these as:

  • 0.0: Marginally respectable/Little to no value

  • 1.0: OK/Somewhat valuable

  • 2.0: Marginal B-journal/Decent field journal

  • 3.0: Top B-journal/Strong field journal

  • 4.0: Marginal A-Journal/Top field journal

  • 5.0: A-journal/Top journal

Evaluation manager’s discussion

The paper presents an important (and impressive) randomized controlled trial (RCT) that evaluates the impact of different timing structures of unconditional cash transfers (or universal basic income) in Kenya.​ There are two important aspects of the paper that make it suitable for evaluation at The Unjournal: the first is the clear linkage to global priorities research and the fact that the authors include individuals from GiveDirectly, an NGO providing UCTs globally and generating an evidence-based approach for giving. The second is that from a knowledge perspective, the findings need to be critically evaluated to better understand what we can learn from RCTs on UBI in the context of Kenya. Given that RCTs are typically considered the “gold standard” of evidence (not just in economics or policy research, but increasingly among policy stakeholders in the Global South).

The study compares three arms: a lump sum transfer, a UBI over two years, and a two-year transfer with the promise of ten years of additional transfers. ​ Each participant received $3,000 in transfers over the study period, with the long-term arm expecting an additional $15,000. ​ The RCT aims to understand how the timing of transfers affects the outcomes, such as work behavior and investment decisions. As Evaluator 1 points out, the study's scale is noteworthy, with nearly 23,000 treated adults and a cost of approximately USD 70 million, making it more comparable to a government program than a standard economics RCT. ​ The findings have implications for policymakers designing benefit structures and organizations like GiveDirectly, who routinely produce evidence at varied levels that could also serve as avenues for future evaluation of applied work at The Unjournal. Evaluating this paper, therefore, gives us an opportunity to better situate such policy evidence in the context of other similar work at GiveDirectly and other non-profits.

Both evaluations agree that the work is likely to make a critical impact on what works in poverty reduction or alleviation within the realm of development economics. The second evaluation especially highlights some of the concerns around causal inference from the claims made by the authors, and provides useful insights on how best to consider these. In particular, the claims around the effects of UBI on well-being and consumption merit some more careful explanation of mechanisms — perhaps the authors could use some aspects of program design (e.g., access to mental health services etc.) to decompose these effects. There are also many other useful comments from the second evaluator on what the authors could meaningfully consider for improving the paper either as part of the current project itself or as future work.

Other notes

One of the evaluators is a field specialist on The Unjournal’s team. However, they were not part of the decision to prioritize this paper for evaluation.

Comments
0
comment
No comments here
Why not start the discussion?