Skip to main content

Evaluation 2 of "Accelerating Vaccine Innovation for Emerging Infectious Diseases via Parallel Discovery"

Evaluation of "Accelerating Vaccine Innovation for Emerging Infectious Diseases via Parallel Discovery" for The Unjournal.

Published onApr 05, 2024
Evaluation 2 of "Accelerating Vaccine Innovation for Emerging Infectious Diseases via Parallel Discovery"
·
key-enterThis Pub is a Supplement to

Abstract

This paper tackles an important global priority area. The main limitations are the premise being restricted to private sector investment, and parameter values used. A societal perspective including (up to 100%) publicly owned efforts, and different reasonable parameter value assumptions, including optimising number of vaccine candidates in relation to expected outbreaks and infections, would alter the conclusions. Calculating cost per QALY gained, and in relation to current healthcare expenditure, would further strengthen the paper.

The abstract above contains the evaluator’s summary of their evaluation of “Accelerating Vaccine Innovation for Emerging Infectious Diseases via Parallel Discovery"[1].

Summary Measures

We asked evaluators to give some overall assessments, in addition to ratings across a range of criteria. See the evaluation summary “metrics” for a more detailed breakdown of this. See these ratings in the context of all Unjournal ratings, with some analysis, in our data presentation here.1

Rating

90% Credible Interval

Overall assessment

60/100

50 - 70

Journal rank tier, normative rating

3/5

2.5 - 3.5

Overall assessment: We asked evaluators to rank this paper “heuristically” as a percentile “relative to all serious research in the same area that you have encountered in the last three years.” We requested they “consider all aspects of quality, credibility, importance to knowledge production, and importance to practice.”

Journal rank tier, normative rating (0-5): “On a ‘scale of journals’, what ‘quality of journal’ should this be published in? (See ranking tiers discussed here)” Note: 0= lowest/none, 5= highest/best”.

See here for the full evaluator guidelines, including further explanation of the requested ratings.

Written report2

My comments are focused on the premise of the paper, and the assumptions and parameter values used – it seems likely that different reasonable assumptions would alter the conclusions of the paper. In this regard the paper would benefit from a societal perspective and an expanded sensitivity analysis, which is explained and reported more clearly in the paper.

  1. The premise of the paper is that “the goal is to create a sustainable business model for addressing EIDs effectively”. If the paper was focused on a broader goal of “addressing EIDs effectively”, i.e., looking beyond private sector business models and also including public sector (government funded) provision (including hypothetical 100% public sector efforts with no private sector involvement, i.e., those driven purely by missions to address EIDs effectively), it might very well come to different conclusions as cost parameter values could be very different including there being no need to produce a profit for private businesses.

  2. Related to the above, an analysis of “market failure” to produce vaccines for EID would be useful. This could follow from Section IIA, which already touches on these issues, and would likely point to the need for public sector efforts, including 100% publicly owned efforts oriented to addressing EIDs and securing human health rather than narrowly to making financial profits.

  3. Following from the above if Net Present Value (NPV) was calculated from the perspective of society (e.g. valuing health gains based on willingness to pay for QALYs or opportunity costs of existing government health expenditure in terms of cost paid per QALY gained – this is around £12,000 in the UK), rather than narrowly from the private sector investors perspective, the results and conclusion would be completely different, and likely highly recommend the vaccines are created and trialled, especially if there is some portfolio optimisation at the outset (see points 5 and 6 on this below).

  4. Page 7: “number of EID outbreaks prevented” – this seems to assume vaccines will prevent an outbreak rather than mitigate the impact of the outbreak? No details are provided on this point, or the “social impact” in general and how it is calculated.

  5. End of page 7, beginning of page 8: “we do not.. .. perform any portfolio optimization” – as the authors acknowledge in the previous paragraph, this (as done for CEPI) is likely to be beneficial, so should really have been done in this paper as another option they look at – one that would likely have more positive results. The authors only crudely look at portfolio size in their sensitivity analysis and find a large effect – why not optimise the portfolio? This could at least be easily done in relation to the expected benefits shown Table 1 (see below point).

  6. Table 1 – how did you decide Nvac for each disease? For example, why are there 23 candidates for West Nile Virus, which has an expected annual probability of outbreak of 10% and average number of infections of 500, whereas there are only 7 candidates for Lassa even though the annual probability of outbreak is 100% and the expected number of infections is 300,000. Changing these would likely have a large impact on your results.

  7. Table 2 narrative: Alpha-tech of 1.2, a 20% increase in PoS (probability of success) of mRNA technology is modest, and in the sensitivity analysis you only vary this to 30% and find decreasing rather than increasing NPV likely because you don’t optimise the portfolio - the numbers of vaccine candidates (Nvac) for each disease (see above point on Table 1) could be chosen in relation to the expected benefits. If this was done and an increased PoS tested it’s possible there could be significant NPV.

  8. Bottom of page 12: why assume the vaccine price is $20 in your main analysis, when you recognise this is less than the list price of all adult vaccines in the US so likely to be an underestimate? Why not use the average list price? In the sensitivity analysis you then talk about prices being above $100 in the US sometimes, and that only $78 is needed for your results to show positive NPV. So this would alter your conclusions, yet in the abstract you don’t mention this sensitivity analysis, or indeed any of your sensitivity analyses, alter your conclusions. The point about low-and middle income countries not being able to afford it seems out of place with the premise of your paper requiring private sector profits.

  9. End of part IV, page 14: “social impact” – how is this defined? No details or justification are given in the methods section.

  10. Results – “the vaccine megafund will require $9.5billion funding from the public sector to generate positive financial value for investors” – this may be a price worth paying for human health security, though going back to points 1-3 above, if the system is wholly publicly owned, the concerns of private investors don’t need to be taken into account, especially considering the public sector could pay for the vaccines ($20 per dose) anyway.

  11. Results: “Using even the most conservative “quality adjusted life year” estimate (e.g., Neumann, Cohen, and Weinstein, 2014), the lives saved and socioeconomic losses avoided by the vaccines far exceed the negative financial value of the megafund.” – It would be good to show estimates of QALYs gained, and methods used to calculate them, and factor them into the overall results e.g. using cost-per-QALY willingness to pay estimates (assuming public healthcare provision) or opportunity cost per QALY gained of current healthcare expenditure (see point 3 above). This would likely make a convincing case for (public) investment in the vaccine megafund i.e. would result in an opposite conclusion of the paper to the current one.

  12. Table 4 footnote – an annual discount rate of 10% is very high. Typically 3-5% is used in health economic evaluations: see Table 1 of https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5999124/. [2]

  13. Sensitivity analysis B: “However, due to cannibalization between vaccines targeting the same EID and the stochastic nature of EID outbreaks, the ultimate revenues increase by a much smaller amount than the investment.” - this result seems to be heavily dependent on the assumptions about cannibalisation, which are not explained in the paper. These should be explained, and varied. Also, if the premise of the paper included a wholly publicly funded system (see points 1-3 above) then the problem of cannibalisation could disappear.

  14. Sensitivity analysis D: what is the assumed pHCT and what is it based on? This is not shown or explained in the paper, though from the appendix (Table S1) it’s apparent HCT is only increased to 30%. If pHCT approaches 100% surely the results would change and there may be positive NPV? This should be shown.

  15. Discussion – the authors note the utility and reduced costs of adaptive and platform trials. These should ideally be modelled in the paper.

  16. Discussion: “A robust and multi-criteria optimization framework is needed to ensure that their value to society is not compromised by optimizing financial returns for the investors.” – as per earlier comments this should ideally be part of the model in this paper.

Evaluator details

  1. How long have you been in this field?

    [Range coded for anonymity: 15-20 years]

  2. How many proposals and papers have you evaluated?

    Around 130

Comments
0
comment
No comments here
Why not start the discussion?