Multi-modal data fusion for prediction of cancer survival
Multimodal fusion focuses on integrating information from multiple modalities with the goal of more accurate prediction. Indeed, models developed to analyze images of pathology and other modalities, such as gene expression, enhance clinical and biological information. The image provides spatial information whereas the gene expression gives the function information. In [1] the proposed framework embedded both image and gene expression data and improves the survival prediction of non small cell lung cancer by leveraging the multimodal data. This approach also introduces a new survival activation maps to help the model interpretability.
[1] Y. Zheng et al., “Graph Attention-Based Fusion of Pathology Images and Gene Expression for Prediction of Cancer Survival,” in IEEE Transactions on Medical Imaging, vol. 43, no. 9, pp. 3085-3097, Sept. 2024
2) The work will focus on how to embedd the gene expression data in a signature to fuse with image information 2) the model interpretability. A comparison between the model based on image versus the model based on multimodal data model on a selected dataset.
