Long Short Term Memory Networks for Lexical Normalization of Tweets

Abstract

Lexical normalization is converting a non-standard text into a standard text that is more readable and universal. Data obtained from social media sites and tweets often contain much noise and use non-canonical sentence structures such as non-standard abbrevlatlons, skipping of words, spelling errors, etc. Hence such data needs to be appropriately processed before it can be used. The processing can be done by lexical normalization, which reduces randomness and converts the sentence structure to a predefined standard. Hence. lexical normalization can help in improving the performance of systems that use user-generated text as inputs. There are several ways to perform lexical normalization, such as dictionary lookups, most frequent replacements, etc. However, VVe aim to explore the domain of deep learning to find approaches that can be used to normalize texts lexically. © 2021 IEEE.