Evaluation Summary and Metrics: "Willful Ignorance and Moral Behavior" for The Unjournal.
We organized two evaluations of the paper: “Willful Ignorance and Moral Behavior”. To read these evaluations, please see the links below.
We asked evaluators to provide overall assessments as well as ratings for a range of specific criteria.
I. Overall assessment: We asked them to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”
II. Journal rank tier, normative rating (0-5):1 On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here.) Note: 0= lowest/none, 5= highest/best.
Overall assessment (0-100) | Journal rank tier, normative rating (0-5) | |
Anonymous evaluator 1 | 90 | 4.0 |
Anonymous evaluator 2 | 88 | 4.7 |
See “Metrics” below for a more detailed breakdown of the evaluators’ ratings across several categories. To see these ratings in the context of all Unjournal ratings, with some analysis, see our data presentation here.2
See here for the current full evaluator guidelines, including further explanation of the requested ratings.3
The paper is professionally written and the experiment is well-designed. The main result, avoiders of a VR animal advocacy message, exhibit larger treatment effects on consumption than information seekers, is novel and important. I find the claims about the dynamics of the treatment effect to be less supported by the evidence. It is more speculative.
This study is an outstanding work that will become a major reference in the empirical literature about information avoidance and meat consumption. First, the authors make here a highly valuable contribution to the economic discipline and, in particular, to the research on dietary transitions by determining the causal impact of information provision on individuals conditional on their a priori willingness to get informed. This is an important research point both for economics in general (where the question of information avoidance has been discussed intensively over the past five years) and for dietary transitions, where researchers and society struggle to induce dietary changes (changes which are needed in light of the environmental, health, and ethical issues arising from large scale meat consumption in developed countries). Second, the authors came up with a lab & field design with real consumption choices, which has been relatively rare in the empirical literature on dietary changes. Following individuals outside of the lab is a key element [for considering] issues like displacement or long-term effects. In addition, looking at effective dietary choices is central, as the attitude-behavior gap in nutrition is a major limiting factor for this research literature. Here, the authors also show that the effect of watching the video on effective consumption choices is short-lived. Third, on the estimation side, the authors propose a neat empirical strategy to elicit the conditional impact (addressing the issue of self-selection), discuss the potential problems at length (e.g., displacement of meat consumption), clearly expose the deviations from their pre-analysis plan, and carefully thought about numerous design issues (e.g., the alternative video). Overall, I am very supportive of this work.
See here for details on the categories below, and the guidance given to evaluators.[1]
Evaluator 1 Joshua Tasoff | Evaluator 2 Romain Espinosa | |||||
|---|---|---|---|---|---|---|
Rating category | Rating (0-100) | 90% CI (0-100)* | Comments | Rating (0-100) | 90% CI (0-100)* | Comments |
Overall assessment4 | 90 | (75, 96) | 88 | (83, 93) | ||
Advancing knowledge and practice5 | 75 | (33, 92) | 6 | 85 | (80, 90) | 7 |
Methods: Justification, reasonableness, validity, robustness8 | 60 | (20, 80) | 9 | 85 | (80, 90) | 10 |
Logic & communication11 | 75 | (33, 92) | 85 | (80, 90) | 12 | |
Open, collaborative, replicable13 | 75 | (33, 92) | 50 | (50, 50) | 14 | |
Real-world relevance 15 | 75 | (33, 92) | 90 | (85, 95) | 16 | |
Relevance to global priorities17 | 75 | (33, 92) | 100 | (100, 100) | 18 |
See here for more details on these tiers.
Evaluator 1 Joshua Tasoff | Evaluator 2 Romain Espinosa | |||||
|---|---|---|---|---|---|---|
Judgment | Ranking tier (0-5) | 90% CI | Comments | Ranking tier (0-5) | 90% CI | Comments |
On a ‘scale of journals’, what ‘quality of journal’ should this be published in? | 4.0 | (3.5, 4.3) | 4.7 | (4.4, 5.0) | ||
What ‘quality journal’ do you expect this work will be published in? | 3.4 | (2.8, 4.0) | 19 | 4.3 | (4.0, 4.6) | 20 |
See here for more details on these tiers. | We summarize these as:
| |||||
Animal welfare advocates argue that consumption of animal products through factory farming leads to massive suffering, and reducing this consumption could substantially reduce this. A range of interventions aimed at influencing individual behavior have been considered, but there is very limited evidence on whether these are successful. Indeed, effective altruism linked animal welfare advocates have moved away from this approach in favor of corporate campaigns.
However, this may be from a lack of evidence rather than evidence that these interventions are ineffective. Of course, there may be heterogeneity; these interventions may be more effective for some audiences than others. It may matter whether these initiatives and ads are targeted towards ‘seekers’, neutral people, or even potential ‘avoiders’. E.g., we could compare ‘push’ vs ‘pull’ marketing, ads targeted towards those showing interest in animals versus more neutral or ‘surprise’ ads etc.
The authors’ work seems to have several rare and valuable strengths:
a credible strategy for differentiating the effects of such information/stimuli on avoiders versus seekers, in a way that seems robust to differential selection issues.
incentivized choices (of food vouchers)
more consequential field evidence (linked to the lab experiment treatments?), with perhaps less potential for demand effects, temporary agreeability bias, etc.
In a general sense, The Unjournal is seeking to stimulate more rigorous social science research relevant to animal welfare, including a focus on consumption attitudes. This is one of our first forays in this direction.
We shared the manager’s notes linked here. Excerpting:
Consistent with The Unjournal’s stated mission, and approach… We are interested in the specific applied and empirical questions, and in the implications of this for animal welfare marketing interventions, and interventions in adjacent areas.
These notes presented a range of fairly specific suggestions for evaluators to consider, as well as Nicolas Treich’s previous notes for the authors.
The authors were aware of this process and happy to have us give feedback on their work, sharing the most recent version of the paper.
One of the evaluators is a member of The Unjournal’s broad team. However, they did not play a decisive role in prioritizing this paper for evaluation.
Evaluation of "Willful Ignorance and Moral Behavior" for The Unjournal.
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
