Scientists at Harvard Medical School have designed a versatile, ChatGPT-like AI model capable of performing an array of diagnostic tasks across multiple forms of cancers. 

The new AI system, described Wednesday in Nature, goes a step beyond many current AI approaches to cancer diagnosis, the researchers said.

Current AI systems are typically trained to perform specific tasks — such as detecting cancer presence or predicting a tumor’s genetic profile — and they tend to work only in a handful of cancer types. By contrast, the new model can perform a wide array of tasks and was tested on 19 cancer types, giving it a flexibility like that of large language models such as ChatGPT.

While other foundation AI models for medical diagnosis based on pathology images have emerged recently, this is believed to be the first to predict patient outcomes and validate them across several international patient groups.

“Our ambition was to create a nimble, versatile ChatGPT-like AI platform that can perform a broad range of cancer evaluation tasks,” said study senior author Kun-Hsing Yu, assistant professor of biomedical informatics in the Blavatnik Institute at Harvard Medical School. “Our model turned out to be very useful across multiple tasks related to cancer detection, prognosis, and treatment response across multiple cancers.”

Training and performance 

The team’s latest work builds on Yu’s previous research in AI systems for the evaluation of colon cancer and brain tumors. These earlier studies demonstrated the feasibility of the approach within specific cancer types and specific tasks.

The new model, called CHIEF (Clinical Histopathology Imaging Evaluation Foundation), was trained on 15 million unlabeled images chunked into sections of interest. The tool was then trained further on 60,000 whole-slide images of tissues including lung, breast, prostate, colorectal, stomach, esophageal, kidney, brain, liver, thyroid, pancreatic, cervical, uterine, ovarian, testicular, skin, soft tissue, adrenal gland, and bladder. Training the model to look both at specific sections of an image and the whole image allowed it to relate specific changes in one region to the overall context. This approach, the researchers said, enabled CHIEF to interpret an image more holistically by considering a broader context, instead of just focusing on a particular region.

Following training, the team tested CHIEF’s performance on more than 19,400 whole-slide images from 32 independent datasets collected from 24 hospitals and patient cohorts across the globe.

The AI model, which works by reading digital slides of tumor tissues, detects cancer cells and predicts a tumor’s molecular profile based on cellular features seen on the image with superior accuracy to most current AI systems. It can forecast patient survival across multiple cancer types and accurately pinpoint features in the tissue that surrounds a tumor — also known as the tumor microenvironment — that are related to a patient’s response to standard treatments, including surgery, chemotherapy, radiation, and immunotherapy. Finally, the team said, the tool appears capable of generating novel insights — it identified specific tumor characteristics previously not known to be linked to patient survival. 

The findings, the research team said, add to growing evidence that AI-powered approaches can enhance clinicians’ ability to evaluate cancers efficiently and accurately, including the identification of patients who might not respond well to standard cancer therapies. 

“If validated further and deployed widely, our approach, and approaches similar to ours, could identify early on cancer patients who may benefit from experimental treatments targeting certain molecular variations, a capability that is not uniformly available across the world,” Yu said.

Overall, CHIEF outperformed other state-of-the-art AI methods by up to 36 percent on the following tasks: cancer cell detection, tumor origin identification, predicting patient outcomes, and identifying the presence of genes and DNA patterns related to treatment response.Because of its versatile training, CHIEF performed equally well no matter how the tumor cells were obtained — whether via biopsy or through surgical excision. And it was just as accurate, regardless of the technique used to digitize the cancer cell samples. This adaptability, the researchers said, renders CHIEF usable across different clinical settings and represents an important step beyond current models that tend to perform well only when reading tissues obtained through specific techniques.

Cancer detection

CHIEF achieved nearly 94 percent accuracy in cancer detection and significantly outperformed current AI approaches across 15 datasets containing 11 cancer types. In five biopsy datasets collected from independent cohorts, CHIEF achieved 96 percent accuracy across multiple cancer types including esophagus, stomach, colon, and prostate. When the researchers tested CHIEF on previously unseen slides from surgically removed tumors of the colon, lung, breast, endometrium, and cervix, the model performed with more than 90 percent accuracy.  

Predicting tumors’ molecular profiles

A tumor’s genetic makeup holds critical clues to determine its future behavior and optimal treatments. To get this information, oncologists order DNA sequencing of tumor samples, but such detailed genomic profiling of cancer tissues is not done routinely nor uniformly across the world due to the cost and time involved in sending samples to specialized DNA sequencing labs. Even in well-resourced regions, the process could take several weeks. It’s a gap that AI could fill, Yu said. 

Quickly identifying cellular patterns on an image suggestive of specific genomic aberrations could offer a quick and cost-effective alternative to genomic sequencing, the researchers said. 

CHIEF outperformed current AI methods for predicting genomic variations in a tumor by looking at the microscopic slides. This new AI approach successfully identified features associated with several important genes related to cancer growth and suppression, and it predicted key genetic mutations related to how well a tumor might respond to various standard therapies. CHIEF also detected specific DNA patterns related to how well a colon tumor might respond to a form of immunotherapy called immune checkpoint blockade.  When looking at whole-tissue images, CHIEF identified mutations in 54 commonly mutated cancer genes with an overall accuracy of more than 70 percent, outperforming the current state-of-the-art AI method for genomic cancer prediction. Its accuracy was greater for specific genes in specific cancer types.

The team also tested CHIEF on its ability to predict mutations linked with response to FDA-approved targeted therapies across 18 genes spanning 15 anatomic sites. CHIEF attained high accuracy in multiple cancer types, including 96 percent in detecting a mutation in a gene called EZH2 common in a blood cancer called diffuse large B-cell lymphoma. It achieved 89 percent for BRAF gene mutation in thyroid cancer, and 91 percent for NTRK1 gene mutation in head and neck cancers. 

Predicting patient survival 

CHIEF successfully predicted patient survival based on tumor histopathology images obtained at the time of initial diagnosis. In all cancer types and all patient groups under study, CHIEF distinguished patients with longer-term survival from those with shorter-term survival. CHIEF outperformed other models by 8 percent.  And in patients with more advanced cancers, CHIEF outperformed other AI models by 10 percent. In all, CHIEF’s ability to predict high versus low death risk was tested and confirmed across patient samples from 17 different institutions.

Extracting novel insights about tumor behavior

The model identified tell-tale patterns on images related to tumor aggressiveness and patient survival. To visualize these areas of interest, CHIEF generated heat maps on an image.  When human pathologists analyzed these AI-derived hot spots, they saw intriguing signals reflecting interactions between cancer cells and surrounding tissues. One such feature was the presence of greater numbers of immune cells in areas of the tumor in longer-term survivors, compared with shorter-term survivors. That finding, Yu noted, makes sense because a greater presence of immune cells may indicate the immune system has been activated to attack the tumor.

When looking at the tumors of shorter-term survivors, CHIEF identified regions of interest marked by the abnormal size ratios between various cell components, more atypical features on the nuclei of cells, weak connections between cells, and less presence of connective tissue in the area surrounding the tumor. These tumors also had a greater presence of dying cells around them. 

For example, in breast tumors, CHIEF pinpointed as an area of interest the presence of necrosis — or cell death — inside the tissues. On the flip side, breast cancers with higher survival rates were more likely to have preserved cellular architecture resembling heathy tissues. The visual features and zones of interest related to survival varied by cancer type, the team noted. 

Next steps

The researchers said they plan to refine CHIEF’s performance and augment its capabilities by:

Conducting additional training on images of tissues from rare diseases and non-cancerous conditions

Including samples from pre-malignant tissues before cells become fully cancerous

Exposing the model to more molecular data to enhance its ability to identify cancers with different levels of aggressiveness

Training the model to also predict the benefits and adverse effects of novel cancer treatments in addition to standard treatments

Authorship: Co-authors included Xiyue Wang, Junhan Zhao, Eliana Marostica, Wei Yuan, Jietian Jin, Jiayu Zhang, Ruijiang Li, Hongping Tang, Kanran Wang, Yu Li, Fang Wang, Yulong Peng, Junyou Zhu, Jing Zhang, Christopher R. Jackson, Jun Zhang, Deborah Dillon, Nancy U. Lin, Lynette Sholl, Thomas Denize, David Meredith, Keith L. Ligon, Sabina Signoretti, Shuji Ogino, Jeffrey A. Golden, MacLean P. Nasrallah, Xiao Han, Sen Yang.

Disclosures: Yu is an inventor of U.S. patent 16/179,101 assigned to Harvard University and served as a consultant for Takeda, Curatio DL, and the Postgraduate Institute for Medicine. Jun Zhang and Han were employees of Tencent AI Lab.

The work was in part supported by the National Institute of General Medical Sciences grant R35GM142879, the Department of Defense Peer Reviewed Cancer Research Program Career Development Award HT9425-23-1-0523, the Google Research Scholar Award, the Harvard Medical School Dean’s Innovation Award, and the Blavatnik Center for Computational Biomedicine Award.