Spatio-temporal prompting network for robust video feature extraction
Author(s)
Type
Conference Paper
Abstract
Frame quality deterioration is one of the main challenges in the field of video understanding. To compensate for the information loss caused by deteriorated frames, recent approaches exploit transformer-based integration modules to obtain spatio-temporal information. However, these integration modules are heavy and complex. Furthermore, each integration module is specifically tailored for its target task, making it difficult to generalise to multiple tasks. In this paper, we present a neat and unified framework, called Spatio-Temporal Prompting Network (STPN). It can efficiently extract robust and accurate video features by dynamically adjusting the input features in the backbone network. Specifically, STPN predicts several video prompts containing spatio-temporal information of neighbour frames. Then, these video prompts are prepended to the patch embeddings of the current frame as the updated input for video feature extraction. Moreover, STPN is easy to generalise to various video tasks because it does not contain task-specific modules. Without bells and whistles, STPN achieves state-of-the-art performance on three widely-used datasets for different video understanding tasks, i.e., ImageNetVID for video object detection, YouTubeVIS for video instance segmentation, and GOT-10k for visual object tracking. Codes are available at https://github.com/guanxiongsun/STPN
Date Issued
2024-01-15
Date Acceptance
2023-10-02
Citation
2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2024, pp.13541-13551
ISSN
1550-5499
Publisher
IEEE Computer Society
Start Page
13541
End Page
13551
Journal / Book Title
2023 IEEE/CVF International Conference on Computer Vision (ICCV)
Copyright Statement
© 2024 IEEE. This ICCV paper is the Open Access version, provided by the Computer Vision Foundation. Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore.
Source
IEEE/CVF International Conference on Computer Vision (ICCV)
Subjects
Computer Science
Computer Science, Artificial Intelligence
Computer Science, Theory & Methods
Imaging Science & Photographic Technology
Science & Technology
Technology
Publication Status
Published
Start Date
2023-10-02
Finish Date
2023-10-06
Coverage Spatial
Paris, France