Multimodal Cancer Prognosis via Clinical Prompt Integration
Original Title: Multimodal deep learning for cancer prognosis prediction with clinical information prompts integration
Journal: NPJ digital medicine
DOI: 10.1038/s41746-025-02257-y
Overview
Survival analysis is a critical component of oncological care, providing the scientific basis for treatment planning and outcome evaluation. While multimodal deep learning has advanced this field by integrating pathology images and genomic data, clinical records are frequently underutilized due to their discrete and low-dimensional nature. This study introduces SurvPGC, a framework designed to bridge this gap by transforming clinical characteristics into high-dimensional embeddings using text templates and a language foundation model. The researchers validated SurvPGC using data from The Cancer Genome Atlas, specifically focusing on liver hepatocellular carcinoma, breast invasive carcinoma, and colorectal adenocarcinoma. The model utilizes a dual-path cross-attention mechanism to facilitate interaction between pathology images, genomic pathways, and clinical prompts. Experimental results indicate that SurvPGC achieves superior performance compared to existing state-of-the-art methods. For instance, the model reached a C-index of 0.701 in both liver and breast cancer datasets, and 0.676 in the colorectal dataset, demonstrating its ability to effectively rank patient risks across diverse cancer types.
Novelty
The primary innovation of this research lies in the methodology used to integrate low-dimensional clinical data with high-dimensional imaging and genomic features. Unlike traditional approaches that simply concatenate risk scores or standardized features, SurvPGC employs natural language processing techniques to create descriptive text prompts from tabular clinical data. By using the CONCH foundation model, these prompts are converted into rich embeddings that allow for deep participation in the model’s training process. Furthermore, the study implements a bidirectional cross-attention fusion strategy. This allows the model to not only exchange information between modalities but also to visualize how different data types influence the focus on specific regions within whole slide images. The results show that clinical information often highlights broader tissue areas, including lymphocytes and stroma, which provides a complementary perspective to the specific focus of genomic data.
Potential Clinical / Research Applications
This research has direct implications for clinical decision support systems. By integrating easily accessible clinical data with complex pathology and genomic information, SurvPGC can provide precise survival estimates at various time points, such as one, three, or five years. In a research setting, the attention visualization tools can help pathologists identify new morphological biomarkers associated with specific clinical profiles or genomic pathways. Furthermore, the framework’s ability to handle diverse data sources makes it a candidate for personalized medicine, where it could assist in tailoring therapeutic interventions based on a comprehensive, multimodal understanding of a patient’s specific disease progression and overall health status.