Multimodal Propaganda Detection in Memes with Tolerance-Based Soft Computing Method

Abstract

This paper presents a tolerance-based near sets-based classifier applied to multimodal propaganda detection task using text and image data originating from Memes. Memes on the internet consist of an image superimposed with text and are very popular in social media. They are often used as a part of disinformation campaign whereby social media users are influenced via a number of rhetorical and psychological techniques known as persuasion techniques. The focus of this paper is on a subtask of the SemEval-2024 Multilingual Detection of Persuasion Techniques Competition in Memes to detect the presence or absence of a persuasion technique. We introduce a multimodal Tolerance Near Sets Classifier (MTNSC) trained on a combination of word embeddings (RoBERTa) and pre-trained image features (ResNet and ResNet-Memes) using the competition data. This work extends our earlier work in the Natural Language Processing domain where a tolerance-based near sets-based sentiment classifier was introduced. The proposed MTNSC achieves a macro F1 score of 70.15% and micro-F1 score of 75.33% on the test dataset demonstrating satisfactory performance of TNS-based classifiers in a multimodal setting. Our findings point to the model’s effectiveness when compared to a few leading submissions based on deep learning techniques. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.