Automatic Video Script Generation based on Natural Language Processing using Deep Learning

Abstract

Detecting their surroundings and describing any image or video scenario in their native language is a very familiar and easy task for humans. However, extracting the temporal aspects of video scene, understanding its context and generating sequence of respective scripts is a challenging job for a machine. Development of methodologies that utilizes the concept of deep learning for applications that generates automatic scripts for the given video scene will be of great importance for people with visual impairment, creation of metadata for images and videos for use by search engines, robot vision systems and film makers and video editors. This article discusses the implementation of Trans- former architecture along with Convolutional Neural Network (CNN) for the effective generation of script for the given video sequence. This work involves the use of CNN to extract the frame features on datasets collected from ActivityNet and YouCookII. After the feature extraction, positional encoding is used for input embedding followed by implementation of Transformer architecture that includes encoder and decoder for effective processing of input frame sequence and respective generation of video scripts. The effectiveness of the work is then tested using Bilingual Evaluation Understudy (BLEU), Recall Oriented Understudy for Gisting Evaluation (ROUGE), Consensus-based Image Description Evaluation (CIDEr) to gauge the precision of text generation and Metric for Evaluation of Translation with Explicit Ordering (METEOR) for a comprehensive evaluation of model as performance metrics.