Evaluation Summary and Metrics: "Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China"
We organized two evaluations of the paper: “Does the Squeaky Wheel Get More Grease? The Direct and Indirect Effects of Citizen Participation on Environmental Governance in China” (2022). Both evaluators rate the paper highly, giving overall ratings of 80 and 90, and stating that the paper ‘should be’ published in a ‘top tier (generalist)’ journal, or a tier just below this (near-top or top-field). To read these evaluations, please click the links at the bottom.
Authors: Mark Buntaine, Michael Greenstone, Guojun He, Mengdi Liu, Shaoda Wang & Bing Zhang
In summary:
The paper evaluated here is the most recent version of the paper that was publicly available from January-August 2023
The authors note they have made substantial revisions in the intervening nine months, although these have not been made public
The authors will consider responding to the evaluations below at some point in the future, noting the changes they have made. They may want to wait until after their paper has been accepted for publication in a traditional journal before they respond.
We reached out to all of the authors in January 2023 to let them know we had chosen their (NBER) paper to be evaluated by The Unjournal as part of our direct evaluation track. The authors did not respond.
It took us over six months to source evaluators for this paper, and get responses. (This is far longer than our targeted turnaround time of approximately 1-2 months).
We contacted the authors in mid-July to share the two completed evaluations of their work, and to give them a chance to respond. The authors replied to our email, noting that they had made “significant revisions to the paper in the intervening nine months, including in response to referees' comments as part of the standard review process.” We suggested some options for their response. In a followup email they stated “after some consideration, we have decided not to offer a detailed response at this stage.” However, they suggested that in the future they would “consider offering a response and explanation about how the paper has changed over time and how it is (or is not) responsive to some of the questions raised by the evaluators”. They suggested that we reach out to them to follow this up.
We recognize that this example poses challenges for The Unjournal’s approach, and we are considering ways to better address this in the future. In particular, we plan to more proactively and persistently reach out to authors to inquire about the status of their papers (and updated versions) before beginning our evaluation process.
See this recent Founder’s Pledge report and summary as a source of some of the points below
Air pollution is a major global source of mortality, particularly in China, South Asia and Africa
This is potentially underfunded (‘neglected’), getting a low amount of ‘funding per death’
Whether more funding and attention is warranted depends on ‘tractability’ (whether funds can have an impact), which depends in large part on the responsiveness and regulatory capacity of governments
This field experiment suggests (but maybe doesn’t nail down) a potentially highly impactful and cost-effective intervention, perhaps involving social media appeals, or other forms of large-scale public social pressure ‘bottom-up’ participation
Furthermore, understanding the nature of the citizen/business/government interaction in China seems highly relevant for a range of other global priorities, including risks from dangerous technologies (perhaps AI, biological ‘gain of function’ research) great-power conflict, and cooperation on global public goods.
“A potentially robust experimental design; the main conclusion (that social media has a key role to play in encouraging compliance with environmental policies) is pretty important; 4th industrial revolution/ social platforms as power”
Both evaluators rate the paper highly, giving overall ratings of 80 and 90, and stating that the paper ‘should be’ published in a ‘top tier (generalist)’ journal, or a tier just below this (near-top or top-field). Both have engaged seriously and substantively, including critiques of the econometrics and identification strategy. The second evaluation also provides a range of large and small specific suggestions for robustness checks, improved exposition, and more.
Eval. 1 : Kubinec | Eval. 2: Anon. | |||||
Rating category | Rating (0-100) | 90% CI (0-100) | Comments (Eval. 1) | Rating (0-100) | 90% CI (0-100) | Comments (Eval. 2) |
Overall assessment | 90 | (85, 95) | 80 | (71, 86) | ||
Advancing knowledge and practice | 90 | (85, 95) | 76 | (68, 83) | ||
Methods: Justification, reasonableness, validity, robustness | 80 | (70, 90) | See comments [in evaluation] | 72 | (63, 81) | |
Logic & communication | 80 | (75, 85) | 67 | (62, 78) | This is more of a personal taste thing - I did not like the way that when reading the paper (particularly Introduction) potential pitfalls jumped at me, and rather than simple say “and this is robust to X concern” or “and as we show, we’re able to exclude X driving these results” I’m left with all of these doubts until I get to quite a bit later in the paper (latter half of results) where you quietly go dispelling most of my concerns… but I’d prefer to not have to store my concerns as I read, and instead be re-assured by you early on that you’ve done this or that check because by and large you have | |
Open, collaborative, replicable | 70 | (60, 80) | Pre-registration missing information | 48 | (42, 68) | While the study says that it is pre-registered, that pre registration does not include any details as to how the study will be analysed, nor anything on the hypotheses that will be tested. Only the experimental design is recorded. It was not apparent where to find the code or data to replicate the study [perhaps not unexpected given it is a WP]. |
Engaging with real-world, impact quantification; practice, realism, and relevance | 90 | (85, 95) | 91 | (78, 94) | ||
Relevance to global priorities | 90 | (85, 95) | 92 | (82, 95) |
Eval. 1: Kubinec | Eval 2: Anon. | |||||
Prediction metric | Rating (0-5) | Confidence (0-5)1 | Comments | Rating (0-5) | 90% CI (0-5) | |
What ‘quality journal’ do you expect this work will be published in? Note: 0= lowest/none, 5= highest/best | 4.8 | (4.0, 5.0) | 4.6 | (3.5, 5) | I think this paper asks a really important question, and then performs a country wide RCT to analyse the answer. While I think my concerns should be addressed - and can be relatively easily with some toning down - this will then likely make the paper less appealing to the very top journals. | |
On a ‘scale of journals’, what ‘quality of journal’ should this be published in? | 4.8 | (4.5, 5.0) | Work has both academic and policy impact and represents state-of-the-art experimental techniques. | 4.3 | (3.7, 4.8) |
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
