Pose2Lang3D: Distilling 3D Reasoning from 2D Skeletons via Language Supervision｜Sai Bhargav Rongali｜Research Insights

収　録: The IEEE/CVF Conference on Computer Vision and Pattern Recognition 2026

要　旨: Despite the progress in 3D human pose estimation, its reliance on expensive multi-view setups and limited dataset availability hinders scalability in real-world applications. We propose a novel framework that distills 3D spatial understanding into a language-aligned representation space using only 2D and 3D pose skeletons rendered as images. Our method learns a shared embedding space where 2D and 3D pose images are projected close to their corresponding natural language descriptions. During training, the model leverages 3D pose supervision to enrich semantic alignment, while at test time, it operates exclusively on 2D poses inferred from real images. This enables high-quality language-based reasoning without an additional computational cost or supervision requirements of 3D pose estimators at inference. Our approach not only reduces reliance on 3D sensors but also demonstrates that 2D pose alone, when trained with 3D-informed language grounding, can achieve rich semantic understanding. Experiments on a newly curated dataset of 80K annotated pose images shows the effectiveness of our method, showing 20.8% and 44.1% improvements over 2D-only baselines and 1.8% and 1.3% improvements over 3D methods in VQA accuracy and BLEU-4 scores respectively.

Sai Bhargav Rongali

収　録: AAAI 2026 Workshop on Assessing and Improving Reliability of Foundation Models in the Real World

要　旨: Vision-language foundation models (VLMs) show strong potential in autonomous driving for scene understanding and decision-making, yet their cross-task performance remains inconsistent. This work presents a systematic study of task interference in VLMs for autonomous driving, revealing a key trade-off: fine-tuning for better perception often degrades planning accuracy. We introduce an evaluation framework that decouples perception and planning to measure interference precisely. Using a multi-source question-answering dataset from diverse driving datasets, we fine-tune state-of the-art VLMs on action descriptions. While fine-tuned models improve decision explanation quality, they exhibit measurable declines in planning compared to zero-shot counterparts. Experiments across multiple architectures confirm this perception–planning trade-off as a general phenomenon driven by attention conflicts and representation divergence. Our findings provide the first empirical validation of foundation model interference in autonomous driving and highlight critical implications for reliable deployment in safety-critical environments.

著　者: Vikrant Nagpure, Tanisha Jain, Rongali Sai Bhargav, Ashwin K Krishna, Kenji Okuma

要　旨: Personalized gaze estimation in in-vehicle settings is crucial for driver monitoring but remains challenging due to individual anatomical differences and environmental variability. Existing approaches often rely on fully supervised fine-tuning, which requires labeled calibration data and limits practical deployment. To overcome this limitation, we propose a Test-Time Personalized Gaze Estimation framework that enables in-vehicle gaze personalization with minimal supervision. Our method builds upon GazeDPTR_V2, which extends gaze estimation to gaze zone classification by leveraging positional features from point projection and visual attributes from images. We introduce a two-stage adaptation strategy: (1) Self-Supervised Test-Time Adaptation, where augmentation-based self-supervision is applied to adapt the model to freely available unlabeled test-time data, facilitated by Model-Agnostic Meta-Learning (MAML) for effective unsupervised adaptation; (2) Weakly-Supervised Personalization, where a small number of gaze zone samples collected from the driver during an initial setup phase are used for fine-tuning. We evaluate our approach on the IVGaze dataset, which provides both gaze direction vectors and gaze zone labels. Experimental results show that our method significantly outperforms baseline models, demonstrating the effectiveness of self-supervised meta-learning and weak supervision for real-world in-vehicle gaze estimation.

Pose2Lang3D: Distilling 3D Reasoning from 2D Skeletons via Language Supervision