Multimodal Disease Classification and Severity analysis Approaches using Machine and Deep Learning

Abstract

Chest X-rays are a common diagnostic tool for pulmonary and cardiac conditions in hospitals because they provide a clear picture of the patient's thorax. With the use of image-to-text radiology report production, medical imaging results may be automatically described in radiology reports. There are a lot of different pieces of patient data that radiologists may access, but most current systems only use the picture data. the objective of developing AI systems with a focus on humans, with the ability to learn radiologists' search habits via their eye movements, with the hope of enhancing DL system categorisation. The goal of this research is to evaluate several multimodal DL architectures in collaboration with trained radiologists to see which ones work best. In particular, this study aims to build strong DL models for medical picture analysis by investigating the integration of several data modalities, such as eye tracking data and patients' clinical data. A multimodal DL model integrating clinical data and chest X-rays (CXRs) was suggested by us. Findings demonstrated that baseline performance was unaffected by directly supplying fixation masks of radiologists' gaze patterns as input. Confine Areas Using R-CNN (Recurrent Neural Networks).