Data-driven multimodal learning towards safer clinical decision support
File(s)
Author(s)
Jha, Sneha
Type
Thesis
Abstract
Multimodal machine learning is one of the most prolific sub-fields of machine learning research.
Multimodal learning methods have made huge strides in a number of domains. Success in using machine learning in clinical and healthcare settings has been relatively less spectacular and impact in real-life workflows has been very slow. A majority of work in the area has been focused on decision-making using a single data modality. This thesis studies issues in the development, evaluation and deployment of ML-based clinical systems that synthesize information from multiple data modalities which is closer to how clinical decision-making operates in the real world.
A majority of existing multimodal learning work, even outside of healthcare has been successful using image and text data. However, structured data of a wide variety is unavoidable in the clinical domain and models that use these efficiently are understudied. Even the success of the text-based models has not completely translated to clinical free-text. This thesis studies the various aspects of leveraging unstructured clinical data in combination with structured electronic health records for critical tasks. In the latter part of the thesis, we study the design of a new explainability method
applicable to clinical data modalities. Inferring the mechanism of information extraction by high-dimensional machine learning models and explaining their automated decision methods is crucial for these models to be safely used in healthcare. It is a well-acknowledged problem with modern machine learning techniques and is significantly harder for under-explored domains and modalities that we study. We conclude this thesis by outlining future directions of research necessary to design clinical machine learning models and decision-making systems with increasing utility, efficiency and safety.
Multimodal learning methods have made huge strides in a number of domains. Success in using machine learning in clinical and healthcare settings has been relatively less spectacular and impact in real-life workflows has been very slow. A majority of work in the area has been focused on decision-making using a single data modality. This thesis studies issues in the development, evaluation and deployment of ML-based clinical systems that synthesize information from multiple data modalities which is closer to how clinical decision-making operates in the real world.
A majority of existing multimodal learning work, even outside of healthcare has been successful using image and text data. However, structured data of a wide variety is unavoidable in the clinical domain and models that use these efficiently are understudied. Even the success of the text-based models has not completely translated to clinical free-text. This thesis studies the various aspects of leveraging unstructured clinical data in combination with structured electronic health records for critical tasks. In the latter part of the thesis, we study the design of a new explainability method
applicable to clinical data modalities. Inferring the mechanism of information extraction by high-dimensional machine learning models and explaining their automated decision methods is crucial for these models to be safely used in healthcare. It is a well-acknowledged problem with modern machine learning techniques and is significantly harder for under-explored domains and modalities that we study. We conclude this thesis by outlining future directions of research necessary to design clinical machine learning models and decision-making systems with increasing utility, efficiency and safety.
Version
Open Access
Date Issued
2025-02-28
Date Awarded
2026-06-01
Copyright Statement
Attribution-NonCommercial 4.0 International Licence (CC BY-NC)
License URL
Advisor
Mayer, Erik
Barahona, Mauricio
Publisher Department
Department of Surgery & Cancer
Department of Mathematics
Publisher Institution
Imperial College London
Qualification Level
Doctoral
Qualification Name
Doctor of Philosophy (PhD)