An iterative MapReduce framework for sports-based tweet clustering

Abstract

In recent years, social media has evolved into a vital source for real-time information. Sports is one of the most popular topics on social media and attracts the attention of users all over the world. However, a large amount of data is generated on a daily basis, making it difficult for the fans to follow the topics of their interest. Clustering of these posts can resolve this issue by retrieving unambiguous and distinct topics. MapReduce is a programming paradigm that is very effective in designing distributed applications that can be deployed on the cloud. Clustering algorithms are generally iterative in nature. The performance gain offered by MapReduce cannot be completely realized by these algorithms due to the inherent architectural bottlenecks associated with iterative tasks. Twister is a MapReduce-based framework designed to minimize these bottlenecks. In this paper, we propose a distributed framework that gathers sports-related tweets and clusters them into distinct topics using the DB-SCAN algorithm customized for Twister. The accuracy of the framework was analysed using the precision-recall scoring mechanism to determine the set of DBSCAN and framework parameters that result in the best set of clusters. The performance of our framework is evaluated based on our clustering results and simulations using the MRSim simulator. We expect that this framework could be used as a model for performing topic detection over generic tweets. We have used the domain of sports to establish the proof of this concept. � 2015 ACM.