Real-Time Emotion Detection in Short Videos through Multimodal Deep Learning Fusion

Resumen

The rapid growth of short-video platforms has significantly increased the volume of multimodal content shared across the internet. These videos typically integrate textual, audio, and visual information, making emotion recognition based on a single modality increasingly ineffective. To address this challenge, this paper proposes a multimodal deep learning framework for real-time emotion detection in short videos by jointly analyzing textual, speech, and visual features.The proposed model employs a multi-head attention mechanism to effectively fuse information from multiple modalities, enabling it to capture complex interrelationships among diverse data sources. In addition, a modal contribution recognition strategy is incorporated to identify the relative importance of each modality in emotion prediction, thereby enhancing feature representation and decision-making. The framework further integrates a multi-task learning approach, allowing the model to optimize multiple related objectives simultaneously, resulting in improved learning efficiency, robustness, and prediction accuracy. The effectiveness of the proposed framework was evaluated using a dataset comprising 900 short videos, with 500 videos used for training and 400 videos reserved for testing. Experimental results demonstrate outstanding performance, achieving an emotion recognition accuracy of 96%, an F1-score of 0.98, and a mean error of only 0.21, indicating highly reliable and balanced emotion classification across multiple categories. Furthermore, the model exhibits excellent computational efficiency, requiring only 2.1 seconds to recognize emotions across 400 test videos, while convergence was achieved within 0.9 seconds after 60 training epochs. These findings demonstrate that the proposed multimodal fusion framework provides a fast, accurate, and scalable solution for real-time emotion recognition in short videos. By leveraging advanced techniques such as multi-head attention, modal contribution recognition, and multi-task learning, the proposed approach significantly enhances the capability of artificial intelligence systems to understand human emotions. The framework has broad application potential in intelligent video recommendation systems, social media analytics, online education, human–computer interaction, virtual assistants, and other emotion-aware AI applications, representing a meaningful advancement in multimodal emotional intelligence research.