key: cord-0999799-vefs8udr authors: Shnaydman, V. title: Simulation model for productivity, risk and GDP impact forecasting of the COVID-19 portfolio vaccines date: 2020-11-04 journal: nan DOI: 10.1101/2020.11.01.20214122 sha: f92079bd0cab541516d1f56240ee14748196ebc6 doc_id: 999799 cord_uid: vefs8udr The paper presents the methodology and modeling results for COVID-19 vaccines portfolio forecasting, including R&D output (rate and likelihood of approvals at a vaccine technology platform level) and manufacturing production output to meet worldwide demand. In order to minimize the time and risk of global vaccination, scaling up of Operation Warp Speed (OWS) and other programs could be very beneficial, leading to increased financing for additional vaccine development programs, in both Phase III clinical trials and in manufacturing. It would also lead to a reduction of the global production time for world vaccination, from 75 months for a baseline scenario to 36 months, reducing potential global GDP loss by as much as US$4.2 trillion (US ~ $1 trillion) when compared to the baseline scenario. COVID-19 vaccine developers are racing against time to develop, test, approve and produce an effective vaccine. Currently more than 200 vaccines are under development 1 versus the 80 to 100 programs that were underway in April 2020. Many programs are still in the preclinical stage, but more than 40 candidates are in clinical trials 2 . It is expected that several vaccines may be approved within months. Each vaccine development program belongs to one of eight technology platforms. Vaccine manufacturers have announced cumulative manufacturing capacity that could produce as many as one billion doses by the end of 2020 and up to 9 billion doses by the end of 2021 [3] . Instead of managing COVID-19 vaccine development programs individually, it was proposed in April 2020 [4] , and in May 2020 [5] that portfolio concepts and methodologies [6] can be applied to the programs. This approach would lead to vaccines. Each block has multiple components or sub-blocks. This paper covers only two sub-models: R&D, and clinical research and manufacturing) due to absence of data about vaccines distribution. (Shnaydman, 2020) . The R&D/clinical research sub-model simulates a vaccine development workflow [3] from preclinical to approval. The simulator utilizes decision tree diagram with binary stochastic outcome (success or failure) [13] . The model [13] was modified to simulate the portfolio of COVID-19 vaccines. Comparing to [4] , the current version of the model presents a more granular view of the vaccine R&D development process at a technology platform level and simulates production of the vaccines. It also incorporates algorithms for simulating interdependence between vaccines and speeding up Phase III trials across the portfolio to mitigate production risk and reduce production time to reach vaccination goals. Modeling capabilities are limited by data availability and data accuracy across the portfolio. Input data for the model was collected from a variety of sources, including databases, academic and trade publications, webinars, podcasts, private communications, and others. Some data reflect a subjective view of the researchers. Therefore, the requirements for data granularity should be balanced with its uniformity across portfolio and sub-models. 1. Portfolio landscape -number of preclinical, Phase I, II, and III candidates (approximately mid-July -August 2020) across eight platforms [1] . 2. Planning horizon is 60 months. Modeling interval is a month. 3 . Platform specific data for each vaccine a. Cycle time for each clinical phase and each vaccine b. Transitional POS at a phase and a platform level c. Integrated monthly manufacturing productivity for each platform d. Vaccine demand for each goal a. Cycle times were significantly compressed due to development urgency. For lead candidates [15, 16] , clinical trial stages are even more compressed and overlapped because of availability of additional funding in order to maximize the speed of development. The result is that much development work is done in parallel and at risk. POS is one of most important drivers of vaccine portfolio productivity. The first release of the model [4] did not differentiate phased POS across vaccine platforms due to absence of data. This assumption was in line with POS data published in several sources at that time, such as [17] [18] [19] [20] [21] . In these papers and reports, integrated POS for all vaccines [18, 19] and vaccines for infectious diseases [20] were calculated based on statistical processing of historical data obtained from previous clinical trials. However, the lack of data needed to calculate POS at a platform level for COVID-19 vaccines portfolio required another approach. The three major approaches used to derive POS are summarized in Table 1 : All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 4, 2020. ; https://doi.org/10.1101/2020.11.01.20214122 doi: medRxiv preprint For accurate simulation of COVID-19 vaccines portfolio, POS elicitation needs to be derived for each platform and each phase of a clinical trial. As the analysis in Table 1 indicates, currently available techniques may not be effective. For example, experts interviewing requires a large number of experts (16 in [8] ) in order to make results statistically significant. Also, COVID-19 vaccines development portfolio is changing so rapidly that making another round of relevant interviews with the same large team of experts could be problematic. Therefore, a new approach was proposed due to lack of historical data: POS elicitation from the development risk prospective. The approach is based on the assumption that the higher the POS, the lower development and regulatory risk, and vice versa. In other words, how can one convert development and regulatory risk into POS? It looks very beneficial to evaluate development risk based on qualitative pairwise comparison of different risks related to global categories such as safety and efficacy at a platform level. Then, platforms are ranked according to risks related to each category, and POS derived according to ranked platforms using Analytic Hierarchy Process (AHP) -a multiple criteria decision making tool that has been widely used by in many applications and verticals [22] [23] [24] , such as planning, resource allocation, and risk management. An example of structured risk analysis for COVID-19 vaccine development using AHP is presented in Figure 2 . The highest level of hierarchy, "Goal -Risk assessment across technology platforms and phases" is comprised of risk categories, such as efficacy and safety. Each category includes several most significant risks. Risk assessment includes two steps. The first step is a pairwise comparison of risk categories at a lexicographic scale from one to nine by an expert or a group of experts to determine weight of each category according to the goal. Pairwise comparison of risk categories generates their weights. For example, a vaccine efficacy of 0.4 and safety of 0.6; the sum should be equal to 1. Then, at the second step, pairwise comparison of risks within each risk category for each platform is conducted. AHP software calculates ranking of each vaccine platform. In addition, experts estimated the minimum and maximum POS for each platform and phase, and the POS for each platform and phase was calculated 3 . Risk assessment in vaccine development is subjective. In order to avoid bias, a group of experts is preferred to make robust assessments [22] [23] [24] . Also, AHP software is able to report inconsistencies in experts' judgements which interviewing lacks. Sensitivity to POS variations for the COVID portfolio simulator was tested in [4] . All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 4, 2020. ; https://doi.org/10.1101/2020.11.01.20214122 doi: medRxiv preprint Figure 2 . Risk hierarchical structure for POS calculation in vaccine development (Shnaydman, 2020) Manufacturing productivity and vaccine demand Several manufacturers have announced their plans to produce about one billion COVID-19 vaccine doses by the end of 2020 and about eight to nine billion doses by the end of 2021 [3, 25] . Not all vaccines are expected to be successful, so production volumes should be adjusted for POS. The model incorporates an integrated network of manufacturing facilities for each platform, and is characterized by its productivity. For example, 25 facilities participate in manufacturing of COVID-19 vaccines [26] [27] [28] . 4 . Algorithms describing interdependence between vaccines. Each platform includes multiple vaccine development programs. Vaccines within a platform have a lot of similarities, but they are still different. Therefore, the risk profile of each vaccine is unique, but the failure or success of a vaccine may have an impact on other candidates within a platform. Vaccine success may reduce risk for other vaccines within a platform and, therefore, increase their POS. Vaccine failure will increase risk for other vaccines, and therefore will reduce their POS. It is assumed in the model that after each successful clinical trial, POS for other trials within a phase increases by x %, and decreases by y % if a clinical trial fails. This algorithm aligns with current practices of portfolio analysis for interdependent drugs, summarized in [14, 17] . Parameters x % and y % are derived based on an expert knowledge 4 . POS may be increased up to maximum value of POS (POS max ) for a platform/phase, and will be decreased to the minimum value (POS min ) for a platform/phase. Incorporation of more complicated algorithms need to be supported by data availability and evidence from industry experts. POS max and POS min were defined in section b "Elicitation of probabilities of success". Robust evidence was not found about vaccines interdependence between platforms 5 . 5. "Operation Warp Speed" scaling up rule 6 All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 4, 2020. ; https://doi.org/10.1101/2020.11.01.20214122 doi: medRxiv preprint The rule permits scaling up of "Operation Warp Speed" (OWS) logistics in case of insufficient portfolio R&D productivity and diversification outside of preselected lead candidates. The rule can be applied to a limited number of qualified candidates due to constrained resources. It will allow overlapping clinical trial phases, and speed up the development and manufacturing process, making "at risk" production for a broader range of development programs. The rule could be applied to selected Phase II candidates, including replacement of failing lead programs. Therefore, second round financing may not be significant as for the original cohort [25] assuming an already developed manufacturing and logistical ecosystem. Authors [8] limited the number of Phase III clinical trials to six. It seems contradictory to the industry practice where all Phase III trials are usually sped up. Implementation of the proposed rule needs to be done gradually depending on portfolio risk profile, available resources, quality of candidates and policies. a. Vaccine approvals within 18 months. Four scenarios were simulated -approvals within (1) 0-6 months; (2) 0-9 months; (3) 0-12 months, and (4) 0-18 months. Simulation results are presented in Figure 3 . (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 4, 2020. ; https://doi.org/10.1101/2020.11.01.20214122 doi: medRxiv preprint F i g u r e 3 l e g e n d : T h e g r a p h s h o w s m a x i m u m n u m b e r o f v a c c i n e a p p r o v a l s ( v e r t i c a l a x i s ) v e r s u s t h e l i k e l i h o o d o f a p p r o v a l s ( L O A ) ( h o r i z o n t a l a x i s ) . F o r e x a m p l e , i n t h e 0 t o 1 8 -m o n t h p e r i o d t h e r e i s a n a l m o s t 1 0 0 % c h a n c e o f a t l e a s t o n e a p p r o v a l , a p p r o x i m a t e l y a n 8 0 % c h a n c e o f m o r e t h a n t w o a p p r o v a l s , a n d a p p r o x i m a t e l y a 5 5 % c h a n c e o f m o r e t h a n t h r e e a p p r o v a l s ( S h n a y d m a n , 2 0 2 0 ) . Modeling results indicate: • For the planning period 0-6 months, LOA of one approval in low (about 15%); • For the planning period 0-9 months, likelihood of zero approvals is about 10%, one approval is about 35%, two approvals is 30%, three approvals is about 21%, and four approvals is less than 5%. LOA for at least one approval is about 90%, LOA of two vaccines approval is 55%, etc. • More favorable results are for planning periods 0-12 months and 0-18 months. There is a good chance that up to three-four vaccines will be approved in 0-18 month period. In further calculations, planning period (0-18) months will be used to analyze portfolio productivity and risk for both R&D/clinical trials and manufacturing. b. Allocation of vaccines approvals across technology platforms. In the baseline scenario, the model forecasts that only vaccines from five platforms out of eight will be approved at the end of 2021 due to the longer duration of clinical trials and the chance of additional funding for remaining platforms. Number of approvals vs. LOA for each platform is presented on Figure 4 . Platforms corresponding with approved vaccines are: Detailed analysis of each platform is presented in many sources, such as [8] . All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 4, 2020. ; https://doi.org/10.1101/2020.11.01.20214122 doi: medRxiv preprint Simulation results presented in Figure 4 indicate that upgrades of manufacturing plants related to five platforms listed above should be prioritized. At the same time, according to [3] , potential manufacturing capacity could be underutilized. How can utilization of available manufacturing capacity be maximized and production risk be minimized? 7 c. Interdependence between vaccines within a platform. Figure 5 analyzes sensitivity of the portfolio productivity to different interdependency rules. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Figure 6 illustrates the POS dynamic for a phase/platform. POS is fluctuating depending on the outcome of previous trials within a platform. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Simulation results indicate that average completion of R&D cycle is around 5-8 months for five prioritized platforms. The R&D cycle time for vaccines in platforms "Live attenuated virus" (#3), "Replicating viral vector" (#6), and "Virus like particles" (#8) exceeds 18 months. Therefore, the manufacturing process will continue for about a year for five platforms to reach a short-term goal of two billion doses. Could short and long-term vaccine production goals be met? What would be the associated risk? Vaccine manufacturers announced cumulative capacity that could produce up to one billion doses in 2020, and up to approximately nine billion in 2021 [3, 25] , but in reality, due to high attrition rates, some candidates may fail and corresponding capacity plans may be altered. Therefore, the model will address several questions, such as: (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 4, 2020. ; https://doi.org/10.1101/2020.11.01.20214122 doi: medRxiv preprint • Would R&D productivity of vaccines pipeline be sufficient to meet worldwide vaccination demand in the shortest time period? • Would manufacturing capacity be sufficient to produce enough vaccine doses worldwide? • Would corresponding production risk of not meeting the goals be tolerable? Production risk is defined as probability of not meeting vaccine demand. The higher demand, the higher the risk of meeting the goal for both end of 2021 (two billion doses) and vaccination of the world (11.2 billion doses). Short-term goal -two billion doses produced by the end of 2021 Figure 7 indicates that the risk of not meeting short-term production goal of two billion doses is relatively high (about 50%). It will be much higher for the world vaccination. How can the production risk be mitigated and the portfolio risk profile be improved in order to meet vaccination goals? Figure 8 presents the dynamics of average vaccine production. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Average dynamic production of 2.2 billion doses means that the risk of meeting the short-term goal is about 50%. How can this risk be reduced? Several potential portfolio mitigation strategies were analyzed in [4] . They included: (1) Increased portfolio size; (2) Compressed cycle times; (3) POS increase by 10%. This paper will expand the number of risk mitigation strategies to guarantee world vaccination in the shortest time and with minimum risk. Simulation results presented on Figure 9 indicate that an increase in manufacturing capacity corresponds with only a marginal increase in vaccine production. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. Therefore, it appears that vaccine manufacturing productivity is not a system bottleneck. Is there a more effective solution? Deeper analysis indicates that R&D portfolio productivity could be a bottleneck. How can R&D productivity be increased? Companies benefiting from the OWS 8 compressed and overlap clinical trials cycle times are based on significant US government funding. These companies became leaders in the vaccine race, and, if their vaccines are successful, they may produce enough doses to meet worldwide demand. At the same time, the risk of not meeting production goals is still significant (see Figure 7) . How can the vaccine portfolio production risk profile be improved? In this paper, the strategy based on scaling of OWS was simulated. According to Figure 10 , number of vaccines approvals significantly increases comparing to the baseline scenario. Portfolio risk profile is also more favorable than in the baseline scenario. b a s e l i n e c a p a c i t y i n c r e a s e 1 0 % c a p a c i t y i n c r e a s e 2 0 % All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 4, 2020. ; https://doi.org/10.1101/2020.11.01.20214122 doi: medRxiv preprint Worldwide vaccines production risk profiles for different risk mitigation scenarios are presented in Figure 11 . V a c c i n e s a p p r o v a l s a n d t h e i r a l l o c a t i o n a c r o s s p l a t f o r m s v s . L O A , 3 0 d e a l s , 0 -1 8 m o n t h s T o t a l s All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 4, 2020. ; https://doi.org/10.1101/2020.11.01.20214122 doi: medRxiv preprint p r o d u c t i o n r i s k , Y a x i s i s c u m u l a t i v e v a c c i n e p r o d u c t i o n , i n m i l l i o n s o f d o s e s . F i g u r e 1 1 i n d i c a t e s t h a t i n " c o l l a b o r a t i v e " s c e n a r i o w i t h 3 0 d e a l s ( f o r 2 0 0 + p o r t f o l i o c a n d i d a t e s ) t h e r i s k o f r e a c h i n g t w o b i l l i o n d o s e s i s c l o s e t o z e r o ; t h e r i s k w i l l b e r e d u c e d f r o m 9 0 % t o a b o u t 3 5 % f o r t h e g o a l o f p r o d u c i n g f o u r b i l l i o n d o s e s (Shnaydman, 2020) . Figure 12 shows the risk adjusted (average) dynamics of the production of multiple vaccines. Production variation increases between months 20 and 40 among the scenarios due to delayed Phase III entrance. The likelihood of worldwide vaccination in 36 months is high as shown in Figure 13 if global policy is focused on portfolio diversification and financing of the most promising candidates beyond original OWS. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. In order to prove effectiveness of proposed COVID vaccines portfolio diversification strategy, the timing of the production of 11.2 billion doses was estimated by crude extrapolation of curves (1) (2) (3) (4) in Figure 12 . In order to do that let's make several assumptions: 1. World GDP = US$ 87.7 trillion in 2019 [29] . 2. Historical world GDP annual growth is about 3.5% 9 [30]. 3. World vaccination will start almost immediately after approvals of vaccines (beginning of 2021). Some vaccination delays are possible, but not significant. 4. World economic growth will be on track if about 70% of the world population will be vaccinated. 5. Four scenarios were analyzed. They are marked on Figure 12 L i k e l i h o o d o f p r o d u c i n g 1 1 . 2 B C O V I D v a c c i n e d o s e s v s . t i m e f o r d i f f e r e n t s c e n a r i o r a t i v e a g r e e m e n t s 3 0 c o l l a b o r a t i v e a g r e e m e n t s ; m a n u f a c t u r i n g c a p a c i t y i n c r e a s e d b y 2 0 % All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 4, 2020. ; https://doi.org/10.1101/2020.11.01.20214122 doi: medRxiv preprint Figure 14 illustrates assessment technique. Figure 14 legend. The diagram illustrates calculation of the World GDP loss for less favorable scenarios than Scenario #4. In 2020, world GDP contracted by about 5% [. In the best scenario, it is assumed that world economy will come back to pre-COVID-19 state in 36 months (world vaccination), and GDP loss=0 comparing to the baseline scenario -75 months (extrapolated) where GDP loss would be maximal (Shnaydman, 2020) . According to the assessment, in order to minimize GDP loss, additional investments should be made to speed up Phase III trials and maximize portfolio productivity. The graph on Figure 15 shows the potential impact of global vaccine development strategies on lost world GDP. All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 4, 2020. ; https://doi.org/10.1101/2020.11.01.20214122 doi: medRxiv preprint 0 Figure 15 legend. The graph shows impact of vaccine development strategies on potential world GDP loss for different vaccines' "weight" in world recovery. For example, even if the world vaccination will contribute 20% to the GDP recovery, GDP loss for Scenarios 1-3 will range from ~0.9T to 2.1T comparing to the "best case" scenario. For 40% world vaccination weight global GDP loss could reach $4.2T (if the "best" scenario is not materialized), for US it could be about $1.05T over three years. X axis -time in months, Y axes -Potential world GDP loss (Shnaydman, 2020 ). ACT Accelerator [31] estimates that $35 billion is required to meet demand in all COVID-19 related products. If several billion dollars will be invested in OWS scale up, return on investment would be significant. The author is very thankful to Michael Quinn. The paper presents a methodology and modeling results for COVID-19 vaccines portfolio forecasting, including R&D output (rate of approvals and their likelihood at a platform level) and manufacturing production output. Methods of analysis include Monte-Carlo portfolio simulation and a technique for elicitation of o t e n t i a l w o r l d G D P l o s s f o r d i f f e r e n t p o r t f o l i o s c e n a r i o s a n d v a c c i n e s " w e i g h t " i n g l o b a l r e c o v e All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. The copyright holder for this preprint this version posted November 4, 2020. ; https://doi.org/10.1101/2020.11.01.20214122 doi: medRxiv preprint 1 probabilities of success (POS) at a platform level. Data was extracted from multiple sources. Simulation results for a baseline scenario indicate that time to produce enough vaccines for world vaccination is significant despite prioritization of a handful portfolio candidates as a part of Operation Warp Speed (OWS). Limited manufacturing capacity and insufficient portfolio diversification contribute to that. In order to minimize the time and risk of world vaccination, scaling up of OWS could be very beneficial including additional financing in both Phase III clinical trials and manufacturing for additional vaccines development programs. It would lead to a reduction in global production time for world vaccination from 75 months for baseline scenario to 36 months, reducing potential global GDP loss up to $4.2 trillion (US -$1 trillion) when compared to the baseline scenario. On pins and needles: Will COVID-19 vaccines 'save the world COVID-19 Vaccines: Tool to Predict and Manage Global Portfolio Productivity and Risk How to pick a real winner in the Covid-19 vaccine race | Financial Times All rights reserved. No reuse allowed without permission. (which was not certified by peer review) is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity COVID-19 Vaccine Predictions: Using Mathematical Modelling and Expert Opinions to Estimate Timelines and Probabilities of Success of COVID-19 Vaccines Covid-19 Vaccine Delivery Will Present Tough Challenge to Cargo Airlines Leave No One Behind: Using a Benefit-Based Advance Market Commitment to Incentivize Development and Global Supply of COVID-19 Vaccines Development and Global Supply of COVID-19 Vaccines Rachel Silverman and Amanda Glassman In the Race to Develop a Vaccine For COVID-19, Is a Pull for R&D Essential or Optional? Industry Drug Development Portfolio Forecasting: Productivity, Risk, Innovation, Sustainability, Commercial Biotechnology Developing Safe and Effective Covid Vaccines -Operation Warp Speed's Strategy and Approach Developing Covid-19 Vaccines at Pandemic Speed. NEJM Clinical development success rates for investigational drugs Estimating the cost of vaccine development against epidemic infectious diseases: a cost minimization study Estimating Probabilities of Success of Vaccine and Other Anti-Infective Therapeutic Development Programs Clinical Approval Success Rates for Investigational Cancer Drugs A Scaling Method for Priorities in Hierarchical Structures The Analytical Hierarchy Process The allocation of intangible resources: the analytic hierarchy process and linear programming The coronavirus vaccine frontrunners are advancing quickly. Here's where they stand. Last updated Warp Speed's Slaoui expects two COVID-19 vaccines this year. BioCentury Developing Safe and Effective Covid Vaccines -Operation Warp Speed's Strategy and Approach NEJM COVID-19: Saving thousands of lives and trillions in livelihoods A Crisis like No Other, An Uncertain Recovery