Faculty Publications
Permanent URI for this communityhttps://idr.nitk.ac.in/handle/123456789/18736
Publications by NITK Faculty
Browse
2 results
Search Results
Item A Novel Approach for Video Captioning Based on Semantic Cross Embedding and Skip-Connection(Springer Science and Business Media Deutschland GmbH, 2021) Radarapu, R.; Bandari, N.; Muthyam, S.; Naik, D.Video Captioning is the task of describing the content of a video in simple natural language. Encoder-Decoder architecture is the most widely used architecture for this task. Recent works exploit the use of 3D Convolutional Neural Networks (CNNs), Transformers or by changing the structure of basic Long Short-Term Memory (LSTM) units used in Encoder-Decoder to improve the performance. In this paper, we propose the use of a sentence vector to improve the performance of the Encoder-Decoder model. This sentence vector acts as an intermediary between the video space and the text space. Thus, it is referred to as semantic cross embedding that bridges the two vector spaces, in this paper. The sentence vector is generated from the video and is used by the Decoder, along with previously generated words to generate a suitable description. We also employ the use of a skip-connection in the Encoder part of the model. Skip-connection is usually employed to tackle the vanishing gradients problem in deep neural networks. However, our experiments show that a two-layer LSTM with a skip-connection performs better than the Bidirectional LSTM, for our model. Also, the use of a sentence vector improves performance considerably. All our experiments are performed on the MSVD dataset. © 2021, Springer Nature Singapore Pte Ltd.Item Video to Text Generation Using Sentence Vector and Skip Connections(Springer, 2023) Mule, H.; Naik, D.Nowadays, video data is increasing rapidly and the need of robust algorithms to process the interpretation of the video. A textual alternative will be more effective and save time. We aim to produce the caption for the video. The most famous architecture used for this is the encoder-decoder (E-D) model. Recent attempts have focused on improving performance by including 3D-CNN, transformers, or structural changes in the basic LSTM units used in E-D. Sentence vectors are used in this work, improving the E-D model’s performance. From the video file, a sentence vector is generated and used by the decoder to generate an accurate description by using previously generated words. Skip connection in the encoder part avoids the vanishing gradients problem. All of our studies use the MSVD and CHARADES datasets. Four famous metrics, BLEU@4, METEOR, ROUGE, and CIDER, are used for performance evaluation. We have compared the performance of BERT, ELMo, and GloVe word embeddings. On experimental analysis, BERT embedding outperformed the ELMo and GloVe embeddings. For feature extraction, pretrained CNNs, NASNet-Large, VGG-16, Inception-v4, and Resnet152 are used, and NASNet-Large outperformed other models. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.