Increasing innovations in the pharmaceutical industry have presented challenges to the conduct of robust, Health Technology Assessment (HTA)-relevant randomised controlled trials. This has led to an increase in the use of non-randomised comparisons between treatments in HTA submissions. These studies carry a risk of bias, limiting their acceptance by HTA bodies and impacting patient access to new treatments.
Quantitative bias analysis (QBA) methods are a range of approaches that utilise information collected outside a primary study dataset to better understand the impact of residual bias on treatment effect estimates. While these methods have great potential to support the use of non-randomised studies, they have received little attention in the HTA setting to-date.
Two recent publications in the Journal of Comparative Effectiveness Research provide insights into their potential value in this setting. The publications are a result of an ongoing collaboration between the real-world evidence team at PHMR, the National Institute for Health and Care Excellence (NICE), academic colleagues at Leiden University, University College London and the London School of Hygiene and Tropical Medicine and F. Hoffmann-La Roche.
Randomised controlled trials (RCTs) represent the gold-standard approach for generating evidence on the comparative effects of treatments to support decision-making by regulators and HTA bodies. However, the conduct of well-powered, unbiased randomised controlled trials (RCTs) is not always feasible, for example due to challenges in maintaining clinical equipoise in trials for very rare diseases. In the absence of RCTs, non-randomised studies (NRS) consisting of single-arm trials, typically coupled with external control arms obtained from historic trials or real-world data sources can be used to compare the effects of treatments. Analyses of this nature have been increasingly submitted to HTA agencies (Patel et al. 2021). These analyses carry a higher risk of bias than RCTs with concerns regarding confounding, measurement error and selection biases often encountered. Due to the challenges in identifying these biases and assessing the impact they could have on estimated treatment effects from NRS, the evidence generated in an NRS are often considered too uncertain for HTA decision makers to utilise. This can limit patient access to new health technologies and disincentivise innovation within the pharmaceutical industry.
QBA are a well-developed suite of methods that allow one to quantitatively explore the potential impact of one or more residual biases on the results of comparative analyses. As such, QBA offer a potentially powerful tool for decision makers to better assess the impact of bias on the results of an NRS and therefore improve the interpretation and utilisation of this evidence in the HTA decision making process. While the methods have been used extensively in the epidemiological literature, QBA has received little attention in the HTA setting (Sammon et al. 2020).
“QBA methods are well developed and established in comparative effectiveness research and important methods in the epidemiologists’ toolbox. They are essential to support the interpretation of NRS in the context of residual biases. It is important that these methods are applied in other fields of research, including HTA applications.”
Rolf H.H. Groenwold (Professor at Leiden University Medical Centre, Department of Clinical Epidemiology and Department of Biomedical Data Sciences)
Our ongoing collaboration with the National Institute for Health and Care Excellence (NICE), academic colleagues at the Leiden University, University College London and London School of Hygiene and Tropical Medicine F. Hoffmann-La Roche seeks to address the underutilisation of these methods in the HTA setting. The first two outputs from this collaboration, have recently been published in the Journal of Comparative Effectiveness Research.
In the first paper titled “Unmeasured confounding in non-randomised studies: Quantitative bias analysis in Health Technology Assessment”, we describe a variety of the QBA methods that have been proposed to adjust for unmeasured confounding in the estimation of treatment effects in non-randomised studies and discuss unique considerations relevant to their application in the HTA setting. The paper reflects on the need to balance methodological complexity with ease of application and interpretation, and the need to ensure the methods fit within the existing frameworks used to assess non-randomised evidence by HTA agencies.
“When conceived and implemented as part of a well-designed and analysed non-randomised study, these methods can play an important role in ensuring estimates of treatment effects from NRS are considered in HTA in a transparent and credible way. The development of methodological guidelines on the application of QBA to the HTA setting should be prioritised to aid the uptake of the methods.”
Manuel Gomes (Associate Professor of Health Economics at the Department of Applied Health Research, University College London)
The second paper titled “Application of Quantitative Bias Analysis for Unmeasured Confounding in Cost-Effectiveness Modelling”, demonstrates the application of QBA methods for unmeasured confounding in a typical HTA setting. That is, we applied two different QBA methods for unmeasured confounding to a simulated non-randomised comparison using synthetic data. The QBA methods were applied under three scenarios representing different levels of knowledge that one might have regarding the unmeasured confounder in practice. The QBA output was then incorporated into a cost-effectiveness model through deterministic and probabilistic sensitivity analyses to demonstrate its potential relevance in markets focused on cost-effectiveness. Ultimately, this paper seeks to provide those working in the HTA community with a tangible example, showcasing the application of QBA methods to HTA-relevant outcomes.
Ongoing work within the collaboration will seek to raise further awareness of QBA within the HTA community, explore the cases in which it has greatest potential to add value and gather stakeholder opinions on best practices in their application.
“HTA bodies like NICE are increasingly considering non-randomised studies to inform their recommendations. QBA methods offer great potential for reducing uncertainty when using non-randomised evidence and supporting committee deliberation but are currently underutilised. The recent publications are intended to raise awareness of QBA methods and stimulate further discussion regarding their potential application within HTA.”
Seamus Kent (Senior Advisor in Data & Analytics, National Institute for Health and Care Excellence)
For more information on our current work in this area please contact [email protected].
Cormac Sammon, Chief Epidemiologist, PHMR
Thomas Leahy, Senior Medical Statistician, PHMR
References
Patel D, Grimson F, Mihaylova E, Wagner P, Warren J, van Engen A, Kim J (2021) Use of external comparators for health technology assessment submissions based on single-arm trials. Value in Health 24 (8):1118-1125
Sammon CJ, Leahy TP, Gsteiger S, Ramagopalan S (2020) Real-world evidence and nonrandomized data in health technology assessment: using existing methods to address unmeasured confounding? J Comp Eff Res 9 (14):969-972. doi:https://doi.org/10.2217/cer-2020-0112
