Abstract
We organized [“two/”three” etc] evaluations of the paper: [Paper title]. [Comments on the evaluations and ratings here … ‘both evaluators rate the paper highly’ etc.] To read these evaluations, please see the links below. [Mention any COI issues in the abstract if they are crucial.]
[Below, we offer a template for hosting Unjournal evaluation summaries in PubPub, version 6, to be linked to the actual evaluations. This template can also be used for sharing with authors (as a draft), for their response, before the full package is released.]
[This template is for the main academic stream; we’re building a slightly different template for our applied and policy stream — see that collection here]
[To proceed…]
Evaluations
1. Name [or ‘anonymous] here, with hyperlink to the evaluation
2. Name [or ‘anonymous] here, with hyperlink to the evaluation
Overall ratings
We asked evaluators to provide overall assessments as well as ratings for a range of specific criteria.
I. Overall assessment: We asked them to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”
II. Journal rank tier, normative rating (0-5): On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here.) Note: 0= lowest/none, 5= highest/best.
FILL IN ACTUAL NUMBERS | Overall assessment (0-100) | Journal rank tier, normative rating (0-5) |
[Name/Anon.] | 67 | 3.5 |
[Name/Anon.] | 43 | 3.4 |
See “Metrics” below for a more detailed breakdown of the evaluators’ ratings across several categories. To see these ratings in the context of all Unjournal ratings, with some analysis, see our data presentation here.
See here for the current full evaluator guidelines, including further explanation of the requested ratings.
Evaluation summaries
[Name/ “Anonymous evaluator 1”]
Reprint summary here, as requested from evaluator.
[Name/ “Anonymous evaluator 2”]
Reprint summary here, as requested from evaluator.
Metrics
Ratings
See here for details on the categories below, and the guidance given to evaluators.
[NOTE: make sure to set the table below as ‘breakout = 75%’ — this is not preserved in cut and paste. Centering may also be lost.]
FILL IN ACTUAL NUMBERS AND COMMENTS | Evaluator 1 [Name/Anon.] | | | Evaluator 2 [Name/Anon.] | | |
|---|
Rating category | Rating (0-100) | 90% CI (0-100)* | Comments (Put longer comments as footnotes) | Rating (0-100) | 90% CI (0-100)* | Comments |
|---|
Overall assessment | 80 | (70, 90) | | 80 | (70, 93) | Shorter comment |
|---|
Advancing knowledge and practice | 25 | (20, 40) | Shorter comment | 90 | (85, 95) | etc. |
|---|
Methods: Justification, reasonableness, validity, robustness | 95 | (85, 97.5) | | 80 | (66, 91) | |
|---|
Logic & communication | 75 | (60, 90) | | 80 | (60,90) | |
|---|
Open, collaborative, replicable | N/A | N/A | | 90 | N/A | |
|---|
Real-world relevance | | | | | | |
|---|
Relevance to global priorities | 60 | (40, 75) | | 95 | (92, 98) | |
|---|
Journal ranking tiers
See here for more details on these tiers.
[NOTE: make sure to set the table below as ‘breakout = 75%’ — this is not preserved in cut and paste]
FILL IN ACTUAL NUMBERS AND COMMENTS | Evaluator 1 [Name/Anon.] | | | Evaluator 2 [Name/Anon.] | | |
|---|
Judgment | Ranking tier (0-5) | 90% CI | Comments (Put longer comments as footnotes) | Ranking tier (0-5) | 90% CI | Comments |
|---|
On a ‘scale of journals’, what ‘quality of journal’ should this be published in? | 3.8 | (2.9, 4.5) | Shorter comment here | 4.0 | (3.5, 4.8) | |
|---|
What ‘quality journal’ do you expect this work will be published in? | 2.3 | (1.8, 3.0) | | 4.2 | (4.0, 5.0) | Shorter comment here |
|---|
See here for more details on these tiers. | We summarize these as: 0.0: Marginally respectable/Little to no value 1.0: OK/Somewhat valuable 2.0: Marginal B-journal/Decent field journal 3.0: Top B-journal/Strong field journal 4.0: Marginal A-Journal/Top field journal 5.0: A-journal/Top journal
| | |
Evaluation manager’s discussion (optional)
Evaluation managers: After the evaluations and author responses are in, you may want to give a brief synthesis and reflection on the research, the evaluations, and the response, considering the implications of these, future directions, etc. You can insert your own judgment here, if you like.
Unjournal process notes (~optional)
Evaluation managers: We usually put a brief discussion of why we prioritized this work and the evaluation process here. Please try to keep this concise: avoid boilerplate, profuse gratitude or flattery. There is no need to report on the parts of this process that worked normally.
Note on versions: [if evaluators considered different versions of the paper]
Why we chose this paper [these headings are optional]
You may want to integrate comments here from the “manager’s notes” (aka ‘bespoke evaluation notes’) previously shared with the evaluators.
How we chose the evaluators
Evaluation process
[Discuss COI issues here if there are any.]
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
