MSF-Mamba: Motion-aware State Fusion Mamba for Efficient Micro-Gesture Recognition

Micro-gesture recognition (MGR) targets the identification of subtle and fine-grained human motions and requires accurate modeling of both long-range and local spatiotemporal dependencies. While convolutional neural networks (CNNs) are effective at capturing local patterns, they struggle with long-range dependencies due to their limited receptive fields. Transformer-based models address this limitation through self-attention mechanisms but suffer from high computational costs. Recently, Mamba has shown promise as an efficient model, leveraging state space models (SSMs) to enable linear-time processing. However, directly applying the vanilla Mamba to MGR may not be optimal. This is because Mamba processes inputs as 1D sequences, with state updates relying solely on the previous state, and thus lacks the ability to model local spatiotemporal dependencies. In addition, previous methods lack a design of motion-awareness, which is crucial in MGR. To overcome these limitations, we propose motion-aware state fusion mamba (MSF-Mamba), which enhances Mamba with local spatiotemporal modeling by fusing local contextual neighboring states. Our design introduces a motion-aware state fusion module based on central frame difference (CFD). Furthermore, a multiscale version named MSF-Mamba+ has been proposed. Specifically, MSF-Mamba+ supports multiscale motion-aware state fusion, as well as an adaptive scale weighting module that dynamically weighs the fused states across different scales. These enhancements explicitly address the limitations of vanilla Mamba by enabling motion-aware local spatiotemporal modeling, allowing MSF-Mamba and MSF-Mamba+ to effectively capture subtle motion cues for MGR. Experiments on two public MGR datasets (i.e., SMG and iMiGUE) demonstrate that even the lightweight version, namely, MSF-Mamba, achieves state-of-the-art performance, outperforming existing CNN-, Transformer-, and SSM-based models while maintaining high efficiency. For example, MSF-Mamba improves Top-1 accuracy by +2.2% and +1.5% over VideoMamba on SMG and iMiGUE, respectively. MSF-Mamba+ outperforms VideoMamba on SMG and iMiGUE, achieving Top-1 accuracy improvements of 2.9% and 3.0%, respectively. The code will be accessible on https://github.com/Leedeng/MSF-Mamba