replicAnt: a pipeline for generating annotated images of animals in complex environments using Unreal Engine
Author(s)
Plum, Fabian
Bulla, René
Beck, Hendrik K
Imirzian, Natalie
Labonte, David
Type
Journal Article
Abstract
Deep learning-based computer vision methods are transforming animal behavioural research. Transfer learning has enabled work in non-model species, but still requires hand-annotation of example footage, and is only performant in well-defined conditions. To help overcome these limitations, we developed replicAnt, a configurable pipeline implemented in Unreal Engine 5 and Python, designed to generate large and variable training datasets on consumer-grade hardware. replicAnt places 3D animal models into complex, procedurally generated environments, from which automatically annotated images can be exported. We demonstrate that synthetic data generated with replicAnt can significantly reduce the hand-annotation required to achieve benchmark performance in common applications such as animal detection, tracking, pose-estimation, and semantic segmentation. We also show that it increases the subject-specificity and domain-invariance of the trained networks, thereby conferring robustness. In some applications, replicAnt may even remove the need for hand-annotation altogether. It thus represents a significant step towards porting deep learning-based computer vision tools to the field.
Date Issued
2023-11-08
Date Acceptance
2023-10-24
Citation
Nature Communications, 2023, 14 (1)
ISSN
2041-1723
Publisher
Nature Portfolio
Journal / Book Title
Nature Communications
Volume
14
Issue
1
Copyright Statement
© The Author(s) 2023. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
License URL
Identifier
https://www.ncbi.nlm.nih.gov/pubmed/37938222
PII: 10.1038/s41467-023-42898-9
Subjects
Animal Experimentation
Animals
Benchmarking
Models, Animal
Semantics
Upper Extremity
Publication Status
Published
Coverage Spatial
England
Article Number
ARTN 7195