REMA-Net: Real-Time Human Action Recognition via Caption-Guided Dual-Stream Attention

Abstract

Real-time human action recognition (HAR) is important for intelligent systems, particularly in human-robot interaction, where robots must respond quickly to human behavior. However, many existing approaches rely on long video sequences or computationally intensive features, limiting their suitability for time-sensitive and resource-constrained environments.

In this paper, we propose REMA-Net, a dual-stream framework that combines visual features from raw RGB frames with semantic representations derived from caption embeddings. Temporal order is incorporated through sinusoidal positional encoding, and a multi-head self-attention mechanism models inter-frame dependencies. A cross-modal knowledge distillation strategy enables information transfer between semantic and visual streams while maintaining a compact architecture.

REMA-Net achieves 96.8%, 73.0%, and 38.2% accuracy on UCF101, HMDB51, and HAA500 datasets, respectively. On an NVIDIA Jetson Orin connected with a TIAGo robot, the model maintains low inference latency, indicating feasibility for embedded robotic platforms. Cross-dataset evaluation without a shared label-mapping protocol shows a similar predictive structure, with confidence and entropy measures remaining within a bounded range across unexplored domains.

Details