Disaster Classification Using Multimodality Techniques by Integrating Images and Text

Abstract

In today’s digital era, the abundance of information shared through social media platforms in times of disaster has emerged as a crucial asset for enhancing disaster response operations. This research initiative was specifically dedicated to enhance disaster categorization by integrating image and tweet text data. The devised model comprises two distinct modules aimed at optimizing the classification process. The primary module focuses on extracting insights from text and images independently using VGG-16 for images, and Bidirectional Long Short-Term Memory and Convolutional Neural Network for texts, subsequently executing the classification task. Conversely, the secondary module is designed to learn the interconnectedness between textual content and images using Contrastive Learning Image Pretraining (CLIP). After this late fusion is used to combine the outcomes of these modules and later softmax classification is used for the classification of the incident into one of seven humanitarian categories thereby enhancing the precision and efficacy of disaster classification. The developed model gives an accuracy of 75% with no data and image augmentations and the result was improved to 93% with different combinations of augmentations. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.