A groundbreaking collaboration between NHS England, academic researchers, and industry partners demonstrates how large-scale, de-identified health data can be leveraged responsibly to enhance healthcare delivery. This work drew on modeling methods developed through the CogStack project, with CogStack providing advice to the team.
The team developed a model from the de-identified healthcare records of 57 million individuals, including hospital admissions, vaccinations, and various medical events, encompassing over 10 billion healthcare interactions spanning more than 40,000 different types of medical events.
Lead researcher Dr Chris Tomlinson (UCL Institute of Health Informatics) said: “AI models are only as good as the data on which they’re trained. So if we want a model that can benefit all patients, with all conditions, then the AI needs to have seen that during training.
“Using national-scale data allows us to represent the kaleidoscopic diversity of England’s population, particularly for minority groups and rare diseases, which are often excluded from research.”
The Foresight generative AI model learns to predict the next likely outcomes based on previous medical events, much like widely used large language models (LLMs), which guess the next word in a sentence based on what they have learned from information available on the internet. Importantly, in this pilot those predictions are limited to COVID-related questions, in line with governance approvals.
Due to the breadth of this secure, population-wide approach, Foresight can also develop insights for minority groups or the growing number of identified rare conditions, often underrepresented in research as they typically form part of smaller datasets.
As Foresight continues to develop, it holds promise for improving both the efficiency and equity of healthcare services across diverse populations. The project highlights how properly governed AI can transform healthcare while maintaining the highest standards of patient privacy and data security.
Researcher Professor Richard Dobson, based at UCL Institute of Health Informatics as well as King’s College London, who is also Deputy Director of the NIHR Maudsley Biomedical Research Centre, said: “This pilot is building on previous research that demonstrated Foresight’s ability to predict health trajectories from data from two NHS trusts. To be able to use it in a national setting is very exciting as it will potentially demonstrate more powerful predictions that can inform services nationally and locally.
What is Foresight?
Foresight represents a significant advancement in healthcare AI, trained on de-identified NHS data within the NHS England Secure Data Environment (SDE). Similar to how ChatGPT predicts text, Foresight functions as an “auto-complete” for medical timelines, predicting future healthcare events based on patterns in historical data. Importantly, these predictions are rigorously validated against real-world outcomes to ensure accuracy.
Collaborative Innovation
Training such sophisticated AI models requires immense datasets and computational resources.
Foresight is a collaboration between NHS England, UCL, King’s College London, National Institute for Health and Care Research Biomedical Research Centres (University College London Hospital, Maudsley), NHS Foundation Trusts (King’s College Hospital, South London and Maudsley), British Heart Foundation Data Science Centre’s CVD-COVID-UK/COVID-IMPACT Consortium, and technology partners AWS, Databricks, and CogStack.
Industry partners like AWS, Databricks and CogStack provided support with computational resources and modeling advice without being able to access NHS data, the AI model, or its outputs. They have no control over the research or its findings.
Funders and oversight bodies include NHS AI Lab, UK Research and Innovation, Medical Research Council, National Institute for Health and Care Research, and Health Data Research UK.
Simon Ellershaw, a PhD researcher at UCL Institute of Health Informatics, said: “Combining the computing resources needed for AI with NHS data has always been challenging, but thanks to the support of our partners we’ve been able to safely and securely apply state-of-the-art AI methods to NHS data at unprecedented scale.”
Looking Forward
This groundbreaking collaboration between NHS England, academic researchers, and industry partners demonstrates how large-scale, de-identified health data can be leveraged responsibly to enhance healthcare delivery, specifically for understanding the impact of COVID-19.
Professor Dobson identifies opportunities for further development. “Currently, the data in this pilot is broad but shallow, and ultimately we’d like to harness the expertise and AI platforms behind Foresight by including richer sources of information like clinicians’ notes, or results of investigations such as blood tests and scans if they become available.”
As Foresight continues to develop, it holds promise for improving both the efficiency and equity of healthcare services across diverse populations.
