3D behavior and neural population dynamics differentiate visual instinctive fear responses of transgenic mice

Abstract

Traditional objective quantitative analysis methods for animal behavior have long necessitated the manual definition of parameters, such as velocity and trajectory, which results in a significant loss of information. With the recent advancements in artificial intelligence and deep learning, the precise tracking and analysis of animal behavior offer the field unprecedented detailed behavioral information.

Here, we investigated the subtle behavior of the mice using the three-dimensional behavioral recording and unsupervised clustering methods in DISC1-N mice (a schizophrenia-associated genetic model) and CCK-KO mice (cholecystokinin knockout) under spontaneous exploration and looming threat paradigms. A novel framework combining 3D skeletal reconstruction with supervised markerless pose estimation with DeepLabCut to reconstruct 3D skeletons, followed by unsupervised behavioral clustering was developed to quantify subtle behavioral differences.

Integration of 3D behavioral tracking, pupillometry, and electrophysiological recordings revealed fine-grained behavioral patterns and neural population dynamics underlying visually guided instinctive fear.

In DISC1-N mice, reductions in risk-assessment behaviors (e.g., sniffing), together with abnormal increases in trotting and rearing during spontaneous exploration were observed, accompanied by hypoactivity of parvalbumin-positive (PV) fast-spiking interneurons in the nucleus accumbens (NAc). Optogenetic activation of NAc-PV neurons partially restored risk-avoidance deficits, implicating NAc-PV dysfunction in schizophrenia-related behavioral inflexibility with reduced information-gathering (sniffing, looking up) and dysfunctional increases in trotting and rearing.

CCK-KO mice exhibited reduced spontaneous sniffing behavior that was accompanied by their augmented defensive responses to threatening stimuli, including heightened escape probability and exaggerated pupillary dilation. Neural population dynamics of CCK-KO mice in latent variable space diverged significantly from WT, highlighting the critical role of cholecystokinin in modulating fear-related neural computations.

Using recurrent switching linear dynamical systems (rSLDS), we found that CCK-KO mice exhibited rigid neural state transitions and strong point attractor dynamics in latent space compared to WT mice, leading to excessive defensive responses to threatening stimuli, while WT mice showed flexible state switching and adaptive behavioral adjustments. In spikeRNN modeling, CCK-KO mice showed a smaller leak factor than WT mice, indicating reduced neural state stability and stronger point attractor dynamics, which may disrupt inhibitory neuromodulation balance and lead to excessive defensive responses and anxiety-related neural mechanism abnormalities.

These findings offer novel evidence for the dysregulation of behavior-neural circuits in psychiatric models (e.g., schizophrenia) and anxiety-related disorders. The methodological integration of 3D ethological analysis with high-dimensional neural decoding demonstrates the unique potential for dissecting the complexity of instinctive behaviors, establishing a cross-species framework for translational research on emotional pathologies.

Date of Award	10 Sept 2025
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Jufang HE (Supervisor) & Liping WANG (External Co-Supervisor)