Abstract

We organized two evaluations of the paper: "Accelerating Vaccine Innovation for Emerging Infectious Diseases via Parallel Discovery". To read these evaluations, please click the links at the bottom.

Key details

Paper: "Accelerating Vaccine Innovation for Emerging Infectious Diseases via Parallel Discovery" [1]

Authors: Joseph Barberio, Jacob Becraft, Zied Ben Chaouch, Dimitris Bertsimas, Tasuku Kitada, Michael Lingzhi Li, Andrew W. Lo, Kevin Shi & Qingyang Xu

Evaluations

1. Evaluation 1, Richard Bruns

2. Evaluation 2, Tim Colbourn

Overall ratings

We asked evaluators to provide overall assessments, in addition to ratings for a range of specific criteria.

I. Overall assessment:1 We asked them to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”

II. Journal rank tier, normative rating (0-5):2 “On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here)” Note: 0= lowest/none, 5= highest/best”.

See “Metrics” below for a more detailed breakdown of the evaluators’ ratings across several categories. To see these ratings in the context of all Unjournal ratings, with some analysis, see our our data presentation here.3

See here for the current full evaluator guidelines, including further explanation of the requested ratings.4

Evaluation summaries

Evaluator 1 : Richard Bruns

This is an advance on the literature, a promising foundation for future research. In its current form I do not find it convincing as a model of the future of vaccine development, and I am very skeptical of its repeated claims that the portfolio will have a 66% chance of preventing a major ‘Disease X’ pandemic, because the paper does not provide enough information about how it simulates the development of these vaccines for a previously unknown pathogen.

However, it provides some insight in its current form (it shows that challenge trials, while potentially helpful, are not sufficient to solve the problem), and there are several minor extensions that could make it much more useful. In particular, Table S1 should be promoted to the main body of the paper, discussed more, and expanded. This table tells us what is more or less likely to make a vaccine portfolio happen, and should be expanded to include more policy choices, such as reforms to decrease the cost of trials.

Evaluator 2: Tim Colbourn

This paper tackles an important global priority area. The main limitations are the premise being restricted to private sector investment, and parameter values used. A societal perspective including (up to 100%) publicly owned efforts, and different reasonable parameter value assumptions, including optimising number of vaccine candidates in relation to expected outbreaks and infections, would alter the conclusions. Calculating cost per QALY gained, and in relation to current healthcare expenditure, would further strengthen the paper.

Metrics

Ratings

See here for details on the categories below, and the guidance given to evaluators.

Journal ranking tiers

See here for more details on these tiers.

Evaluation manager’s discussion (by David Manheim)

The question of how to fund vaccines for novel diseases is critical for preventing or mitigating future pandemics, and given the current speed of vaccine development and the relative lack of investment, commercial approaches for funding seem important. Given that, this paper targets an important question. That said, it is unclear to me, and the reviewers [evaluators], whether the specific approach is among the most promising, and the evaluation is of a fairly narrowly defined approach. At the same time, most of the points investigated to come to the conclusion seem broadly important to any diversified approach for pre-development of vaccines.

The reviews [evaluations] primarily highlight the limitations, because the value of the paper's investigation overall is obvious. At the same time, the limitations of the paper as it stands make it far less valuable than it could be, as the reviews note. For that reason, it is disappointing that the authors have not yet engaged, because of the room to so greatly improve the impact of the papers' investigation.