InteBOMB: Integrating generic object tracking and segmentation with pose estimation for animal behavioral analysis

Over the past years, numerous animal behavior quantification methods have opened up a new field of computational ethology and will make it possible to establish general-purpose methods for fully automated behavior analysis. Existing state-of-the-art multi-animal pose estimation workflows adopt tracking-by-detection for either bottom-up or top-down approaches, which have to be retrained for diverse animal appearances. Our proposed InteBOMB workflow integrates generic object tracking into top-down approaches, not needing to know the animal as a priori and maintaining the comparative advantage of generalization. Specifically, our main contribution includes two strategies for tracking and segmentation in laboratory scenes and two techniques for pose estimation in natural scenes. The “background enhancement” strategy calculates foreground-background contrastive losses, building more discriminative correlation maps. The “online proofreading” strategy stores human-in-the-loop long-term memory and dynamic short-term memory, updating visual features of objects actively. The “automated labeling suggestion” technique reuses the visual features saved during tracking, selecting representative frames to label the training sets. The “joint behavior analysis” technique also brings these features to combine with data from other modalities, extending the latent space for behavior classification and clustering. Notably, we collected six datasets of mice and six datasets of non-human primates to benchmark laboratory and natural scenes, which evaluated our zero-shot generic trackers and the high-performance joint latent space, with an average of 24% and 21% improvement across datasets, respectively.