## Repository structure notebooks/ 01_click_detection.ipynb 02_CQT_extraction.ipynb 03_click_train_clustering.ipynb 04_spatial_localization.ipynb 05_video_projection.ipynb utils.py **Note on architecture:** Common mathematical functions and hardware constants for the 3-hydrophone array (such as the local sound speed and sensor pair configurations) are centralized in the `utils.py` module to ensure consistency across the pipeline. ## Authorship The core mathematical functions and foundational signal processing algorithms utilized in this pipeline were originally developed by Walter M.X Zimmer: * (c) Walter M.X. Zimmer (2025) * Modified and expanded by Lara Berkenbaum and Hervé Glotin (2025, 2026) for the specific multimodal tracking framework described in the manuscript. This code constitutes the CIAN Tool Kit developped for the study: Berkenbaum, L., Glotin, H., Sarano, F., Zimmer, W.M.X., Sarano, V., Adam, O., Giraudet, P. (2026). "Audiovisual diarization of overlapping click trains in sperm whale (Physeter macrocephalus) vocal sparring using a three-hydrophone array," J. Acoust. Soc. Am. --- ## Data pipeline and unsupervised clustering workflow To ensure full transparency regarding the repository's data flow, please note the distinction between the data conditioning, the automated acoustic clustering, and the final individual assignments: 1. **Data conditioning and filtering:** Manual validation and exclusion criteria are applied upstream in the `_validated_clicks.csv` files. Specific data filters are documented via the `excluded_reason` column. Module 02 natively respects this column, selectively computing CQT patches only for valid, retained signals. 2. **Unsupervised acoustic clustering (Module 03):** this script relies on spectro-temporal tracking of the CQT patches to group individual clicks into coherent click trains. It operates completely blind to the visual ground truth and outputs the `predicted_clust` column, which represents unlabeled acoustic fragments. Note on DERclust computaion. Module 03 documents the automatic tracking method; its raw output is a preliminary partition. All diarization metrics (DERclust) reported in the manuscript are computed from the provided `*_validated_clicks.csv` files, comparing the `predicted_clust` column (automatic clustering label) against the `track` column (manually validated ground truth), scene by scene. 4. **Manual optical attribution:** as described in the manuscript (Sec. II.F), the attribution of these acoustic trains to specific named individuals is an independent, human-driven process. It is achieved through manual, frame-by-frame visual inspection, intersecting precise acoustic localizations with the optical target. 5. **Final optical rendering (Module 05):** this human-validated attribution produces the `final_ID` column found in the `localization_dataset.csv` files. Module 05 ingests this finalized dataset to render the optical validation sequences. **Data dictionary for validated clicks (`*_validated_clicks.csv`)** One row per detected and manually validated click (true positives). This file is the entry point of both processing chains and ensures full traceability of inclusion/exclusion criteria. * `index`: identifier of the click within the session. * `session_id`: session identifier (e.g., 085452). * `scene_id`: interaction scene the click belongs to, following the manuscript numbering; `no_scene` if the click falls outside any defined scene. * `T_H0`: arrival time on the reference hydrophone (in seconds). * `passed_env_filter`: flag (1/0) indicating whether the click passed the adaptive envelope tracking filter. Computed on detected true positives only; minor count differences with the manuscript totals reflect clicks missed at detection (false negatives), which are not present in this file. * `focal_scene`: flag (1/0) indicating whether the click belongs to one of the focal interaction scenes. * `is_focal`: flag (1/0) indicating whether the click is attributed to a focal emitter (or, in session 085315, retained as a focal candidate submitted to localization). * `predicted_clust`: raw, automatically generated cluster identifier produced by Module 03. Empty (`NaN`) if the click was not clustered; `-1` if it was a clustering candidate left unassigned (counted as an omission by the Diarization Error Rate clustering evaluation). * `track`: the final, human-corrected acoustic track following manual review (Level 2). This is the manual reference against which `predicted_clust` is evaluated. * `excluded_reason`: categorical string documenting the scene-based mask applied by module 02; empty (`NaN`) if the click enters the notebook clustering. The mask is stricter than the selection used for the reference partition (see the note on DERclust computaion above. The closed list of motifs includes: * `out_of_focal_scenes`: click falling outside any defined scene window. * `non_focal_scene`: click belonging to a defined scene that is not focal (Scenes 10-11 in session 085315 and scene 12 in session 085452). * `non_sparring`: click belonging to distinct acoustic patterns (e.g., codas). **Data dictionary for spatial and optical validation (`localization_optical_dataset.csv`)** This master file consolidates the outputs from the spatial localization chain (Module 04) and the clustering/attribution process. It serves as the definitive input for the final video rendering and spatial error evaluation (Module 05). Metadata and temporal synchronization: * `session_id`: session identifier. * `scene_id`: interaction scene the click belongs to; `no_scene` if the click falls outside any defined scene. * `T_H0_loc`: arrival time on the reference hydrophone (in seconds), extracted using the strict detection thresholds required for localization. This is the join key linking each row back to the raw `_detections_strict.csv` and `localizations.csv` files (matched on `T_H0` within the same `session_id`). * `T_H0_clust`: reconciled arrival time from the clustering chain (permissive detection), validating the temporal match between the two parallel acoustic processing branches. Identity and attribution: * `track`: the final tracks, human-corrected acoustic track number. * `final_ID`: the name of the assigned focal individual (e.g., Ali or Daren). Clicks from scenes 5 and 6 that were localized but could not be definitively assigned are explicitly labeled as `unresolved`. * `focal`: flag (1/0) indicating whether the click is attributed to a focal emitter. * `optical_subset`: flag (1/0) indicating whether this click belongs to the filtered subset used for final optical projection and error analysis. Acoustic Localization (Module 04 metrics): * `B01_samples`, `B02_samples`, `B12_samples`: inter-sensor cross-correlation delays (Time Difference of Arrival) estimated with sub-sample precision. * `AZ_C_deg`, `EL_C_deg`: acoustic azimuth and elevation angles (in degrees), mathematically rotated and corrected to align with the camera's orthonormal optical reference frame. * `energy_dB`: click energy (in dB). * `loop_error_cm`: geometric loop-closure error (in cm) used to filter spatial aberrations. * `residuals_cm`: spatial residuals (in cm) indicating the discrepancy between measured delays and theoretical delays of the reconstructed vector. Optical Projection (Module 05 metrics): * `ref_anat_x_fov`, `ref_anat_y_fov`: position of the manually tracked anatomical reference (distal end of the junk) in the image plane, expressed in FOV degrees within a centered frame (X ∈ [-59, +59], Y ∈ [-34.5, +34.5]). * `acoustic_x_fov`, `acoustic_y_fov`: linear projection of the acoustic angles into the same 2D optical image plane, allowing for direct calculation of the projection error. --- ## Module 01 - Click detection and TDOA extraction Module 01 detects sperm whale click transients on the raw recordings and extracts the inter-hydrophone TDOA delays. It serves as the entry point of the pipeline, feeding both the clustering and spatial localization chains. **File naming convention** Raw audio files must follow the convention `YYYYMMDD_HHMMSSUTC_Vxx.wav`. The six-digit time field (HHMMSS) acts as the session identifier, which is dynamically extracted to match detections, localizations, and video sequences throughout the pipeline. **Hardware and array configuration** The functional subarray comprises three heterogeneous hydrophones: channels H0 and H1 (SQ26) define the horizontal axis, while channel H2 (C75) is positioned lower and forward to form an inclined plane required for 3D resolution. In the raw 5-channel audio files, these physically correspond to Python indices `[0,1,4]`. The script selects these indices accordingly, with channel 0 (SQ26) acting as the reference hydrophone. **Pre-processing and calibration** A fourth-order Butterworth high-pass filter (5-kHz cut-off) attenuates low-frequency oceanic noise prior to TKEO-based transient detection. To account for heterogeneous sensor sensitivities, a per-channel empirical correction factor of `[1,1,-3]` is applied. Exact hardware sensitivities and corresponding correction parameters are detailed in Supplementary Material S1-S2. **Two-stage detection strategy** Detection is executed twice using asymmetric thresholds to produce two distinct files per recording: * `_detections_permissive.csv` (Thresholds: SQ26 (H0/H1) = 40, C75 (H2) = 30): maximizes track continuity to feed the unsupervised clustering chain (Modules 02-03). * `_detections_strict.csv` (Thresholds: SQ26 (H0/H1) = 80, C75 (H2) = 55): enforces a near-zero false-alarm rate to feed the spatial localization chain (Module 04). These two detection branches are independent and are reconciled later via temporal matching. **Outputs and data dictionary** Each output CSV contains the following columns per detected click: * `T_H0`, `T_H1`, `T_H2`: arrival times per hydrophone (in seconds). * `B01_samples`, `B02_samples`, `B12_samples`: inter-sensor cross-correlation delays (in samples). * `energy_dB`: click energy. * `loop_error_cm`: geometric loop-closure error. * `fs_hz`: sampling rate. --- ## Module 02 - Constant-Q Transform (CQT) extraction Module 02 computes a CQT patch for each validated click retained for clustering. These time-frequency patches are the acoustic signatures used by the unsupervised clustering (Module 03). Scalar spectral descriptors (centroid, bandwidth, entropy, energy) are also computed and stored for reference but are not used by the clustering. **Data filtering and computational optimization** The `excluded_reason` column documents the scene-based exclusion mask applied by Module 02 when computing CQT patches. This mask is stricter than the label-based selection originally used to produce the reference clustering partition, and is not intended to reproduce that partition exactly; the reference partition is provided directly in the `predicted_clust` and `track` columns of the validated files (see Module 03). **Data flow and traceability** To ensure a robust and unbroken data chain between the validated dataset and the clustering algorithm, this module unifies the click identification using the `index` column. It also actively propagates the arrival time (`T_H0`) alongside each click's CQT patch and identifier. **Note on pre-processing decoupling** Module 02 intentionally uses a different pre-processing strategy from Module 01. This preserves the full broadband spectral content required for accurate Constant-Q Transform (CQT) analysis. Any minor temporal discrepancies introduced by decoupling the spectral extraction from the TKEO transient detection are computationally absorbed by the 4 ms local energy peak realignment window. **Outputs and data dictionary** The module outputs a pickle file containing the CQT patches indexed by click (`_CQT_patch_H0.pkl`) and a CSV file (`_CQT_feat_H0.csv`) containing the following columns: * `cqt_centroid`, `cqt_bandwidth`, `cqt_entropy`, `cqt_energy`: scalar spectral descriptors, provided for reference (not used by the clustering, which operates on the full CQT patches stored in the pickle). * `index`: the identifier of the click. * `T_H0`: the arrival time on the reference hydrophone, propagated for the tracking algorithm. * `status_cqt`: extraction status flag (`"ok"` or `"empty"`). --- ## Module 03 - Click train spectral clustering Module 03 relies on spectro-temporal tracking to group individual clicks into coherent click trains. It utilizes a tracking algorithm with short-term and long-term template memory to group unlabelled acoustic fragments, compensating for temporal jitter via cross-correlation shifts. **Data flow and input filtering** The clustering algorithm operates completely blind to the visual ground truth. It ingests the temporal metadata (`T_H0` from `_CQT_feat_H0.csv`) and the acoustic signatures (the CQT patches from `_CQT_patches_H0.pkl`) propagated by Module 02, using the unified `index` column to match features with their corresponding CQT patches. The notebook processes the clicks retained by the `excluded_reason` mask and is provided to document the tracking method. Because this mask is stricter than the selection used fo the reference partition, the notebook output is a preliminary partition and is not expected to reproduce the reference `predicted_clust/track` labels verbatim. **Important note regarding session 085315 (Scenes 5 & 6)** For this specific session, clicks belonging to Scene 5 and 6 are correctly ingested and clustered by Module 03 (as they do not possess an `excluded_reason`). However, because their corresponding optical attribution could not be definitively confirmed, they are designated as `unresolved` in downstream modules. Consequently, these specific clusters are explicitly separated from the final focal performance evaluation (e.g., Table II in the manuscript), appearing only as unconfirmed in Table IV. **Outputs and data dictionary** The module outputs a clustered dataset (`_click_clustering.csv`) containing the following key column appended to the input data: * `fragment_id`: preliminary track fragment identifier, before the a posteriori fusion phase. * `clustered_merged`: automatic cluster identifier after the fusion phase. --- ## Module 04 - Spatial localization (TDOA to angles) Module 04 computes the 3D spatial origin of each click based on the strict TDOA delays extracted in Module 01 (thresholds defined in Sec. II.C). **Methodology and geometry** The localization relies on a constrained two-dimensional directional approach to resolve the spatial coordinates. The OPALE array geometry corresponds to the coordinate matrix provided in Supplementary material S1. **Cone boundary folding and normal reconstruction** To reconstruct the normal component, the algorithm postulates a frontal half-space. Directions marginally exceeding the array's resolvable cone are folded back to the cone boundary. This mathematical regularization retains these clicks rather than discarding them, assuming the excess stems from measurement noise on near-plane sources. **Outputs and data dictionary** The module outputs `_localizations.csv` containing the following columns: * `AZ_G_deg`, `EL_G_deg`: azimuth and elevation angles in the antenna's geometric reference frame (and their radian equivalents, suffixed `_rad`). * `AZ_C_deg`,`EL_C_deg`: azimuth and elevation angles rotated and corrected to align with the camera's orthonormal optical reference frame (and their radian equivalents, suffixed `_rad`). * `residuals_cm`: spatial residuals for TDOA quality assessment. **Note on traceability** Module 04 localizes all raw detections from `*_detections_strict.csv`. To reproduce the exact validated subsets and final counts presented in the manuscript, filter these results using the master file `localization_optical_dataset.csv` (matching `T_H0` in the raw files to `T_H0_loc` in the master file, within the same `session_id`). --- ## MODULE 05 - Video projection and optical validation **Note on optical validation and GIF generation** Module 05 is intentionally designed to output isolated GIF sequences rather than full, continuous video files. This deliberate design choice provides a method to audit the temporal synchronization and the linear angle-to-pixel projection logic on a strict *click-by-click* (frame-by-frame) basis. By isolating these sequences, the pipeline ensures methodological transparency and allows for a precise visual verification of how the acoustic spatial coordinates are mapped against the optical ground truth. Crucially, this output includes Scenes 5 and 6, purposefully illustrates localized acoustic trains that could not be attributed to a named individual. **Data availability and output directory** Due to the file size of the raw high-resolution recordings, the raw videos are not included in the public repository, and this module therefore cannot be re-executed as-is; it is provided to document the click-by-click projection logic. Whereas the code renders one GIF per acoustic track, the `03_videos` directory provides the finalized deliverable: nine rendered .mp4 sequences, one per focal interaction scene (matching the nine focal scenes of the manuscript), each showing the click-to-individual assignement for every individual present in that scene, including the unresolved scenes 5 and 6.