Evaluation of "Accelerating Vaccine Innovation for Emerging Infectious Diseases via Parallel Discovery" for The Unjournal.
This is an evaluation of "Accelerating Vaccine Innovation for Emerging Infectious Diseases via Parallel Discovery". [Extract from the evaluator’s summary follows.] This is an advance on the literature, a promising foundation for future research. In its current form I do not find it convincing as a model of the future of vaccine development, and I am very skeptical of its repeated claims that the portfolio will have a 66% chance of preventing a major ‘Disease X’ pandemic, because the paper does not provide enough information about how it simulates the development of these vaccines for a previously unknown pathogen. However, it provides some insight in its current form (it shows that challenge trials, while potentially helpful, are not sufficient to solve the problem), and there are several minor extensions that could make it much more useful.
We asked evaluators to give some overall assessments, in addition to ratings across a range of criteria. See the evaluation summary “metrics” for a more detailed breakdown of this. See these ratings in the context of all Unjournal ratings, with some analysis, in our data presentation here.1
Rating | 90% Credible Interval | |
Overall assessment | 40/100 | 20 - 60 |
Journal rank tier, normative rating | 2/5 | 1 - 3 |
Overall assessment: We asked evaluators to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”
Journal rank tier, normative rating (0-5): “On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here)” Note: 0= lowest/none, 5= highest/best”.
See here for the full evaluator guidelines, including further explanation of the requested ratings.
This is an advance on the literature, a promising foundation for future research. In its current form I do not find it convincing as a model of the future of vaccine development, and I am very skeptical of its repeated claims that the portfolio will have a 66% chance of preventing a major ‘Disease X’ pandemic, because the paper does not provide enough information about how it simulates the development of these vaccines for a previously unknown pathogen.
However, it provides some insight in its current form (it shows that challenge trials, while potentially helpful, are not sufficient to solve the problem), and there are several minor extensions that could make it much more useful. In particular, Table S1 should be promoted to the main body of the paper, discussed more, and expanded. This table tells us what is more or less likely to make a vaccine portfolio happen, and should be expanded to include more policy choices, such as reforms to decrease the cost of trials.
The doses per outbreak should be a variable, ideally one different for each disease, rather than fixed at 10 million. For each known emerging infectious disease, there is a probability that it will become a major global pandemic, but also a probability that it will be smaller or can be contained. The following should be researched, possibly via expert elicitation:
For each disease, what is the 90% CI of the number of vaccine doses that would be required to contain the outbreak with a ring vaccination approach? For example, the 2018 Kiva Ebola outbreak was contained with about 100,000 vaccines.
What is the probability that containment fails and the disease becomes a global pandemic?
Much, perhaps most, of the returns are driven by the ‘Disease X’ vaccine sales. The paper does not fully explain how this is modeled, but from what I can see, I suspect that it is wrong to include it in the portfolio. At minimum, it should include a sensitivity analysis where the Disease X vaccines are removed from the results.
It seems from Page 13 that the Disease X vaccine is not developed to completion of phase 2 in the absence of an outbreak. However, the paper never states how and when the vaccine is developed, and how it relates to the rest of the portfolio. I suspect that there is a special logic for an accelerated trial after the Disease X outbreak starts. But if this is the case, then Disease X should not be included in the portfolio, because it is a completely separate program and research track.
A better modeling approach would be to use a viral families approach, as described by CSET and CHS:
https://centerforhealthsecurity.org/sites/default/files/2022-12/180510-pandemic-pathogens-report.pdf [2]
Disease X is likely to come from one of a small number of vaccine families. If there is an approved vaccine for another virus in that family, it will be easier to develop one for Disease X. For example, if Disease X is a paramyxovirus, and a Nipah vaccine has been approved, that Nipah vaccine might be effective against Disease X, and it will be easier to develop and produce a new vaccine specifically for Disease X.
The simulation framework for Disease X should be:
If Disease X happens, randomly choose which viral family it comes from.
If there is no vaccine approved or in the pipeline for any disease in that family, there is no cash flow from the portfolio, because any development will be de novo.
If there is a vaccine in development, or approved, for a disease in that family, an expedited development and trial can begin. The development of the mpox vaccine can be used as a case study for this.
There is a probability that the existing vaccine can be used immediately for Disease X (likely with a lower effectiveness, but still providing cash flow and social value).
The number of vaccines sold is variable. Some Disease X outbreaks may be large, but others may be small, like mpox. (Expand the definition and probability of ‘Disease X’ to account for this larger definition, such that there is still a 1% chance of a major outbreak.)
The cost per dose is key to profitability, but [this is] modeled with little rigor. In the primary specification, they assume a fixed $20 per dose based on very little information. They could easily extend the model to market-based pricing, where a company could charge more if it has a monopoly and less if there is competition. The simulation should be upgraded so that if only one vaccine candidate has been successfully researched, the price is higher. (Even so, Table S1 shows that even with a price of $100 per dose, the returns are not high enough to compensate investors for the risk in the absence of other reforms.)
The main takeaway from this paper is that we need regulatory reforms to make vaccine clinical trials cheaper. However, the paper authors have very little discussion of this, and tend to treat clinical trial costs as a law of nature rather than a policy choice.
The ‘Average Number of Infections’ column in Table 1 is extraneous and misleading, having no impact on the simulation. They always assume that 10 million doses will be sold for all vaccines.
The bin size on Figure 3 is much too large, and the histograms reveal very little.
There are many virus families that are candidates for a Disease X, such as picornaviruses, that are not associated with the Emerging Infectious Diseases described in this paper. However, they are associated with seasonal respiratory viruses, and there may be a market for them. For example, many schoolteachers might like to take an annual rhinovirus vaccine. The model should include them.
Vaccines in these virus families will, if successful, generate some annual cash flow, while also having a chance of being useful against Disease X.
The cost per clinical trial should be a variable input, and Table S1 should show profitability with lower trial costs. While there may be some reforms that simply improve efficiency, other reforms may generate tradeoffs in the form of slightly lower safety and efficacy, and the costs and benefits of these expedited and cheaper trials can be modeled. As a first step, however, this simulation can easily show how much cheaper clinical trials would have to be for the portfolio to be profitable.
How long have you been in this field?
[Range coded to preserve anonymity: 10-15 years]
How many proposals and papers have you evaluated?
Several dozen
Evaluation of "Accelerating Vaccine Innovation for Emerging Infectious Diseases via Parallel Discovery" for The Unjournal.
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
