3D-VCDHallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decoding

Makanjuola Ogunleye^●^*, Eman Abdelrahman^●^*, Ismini Lourentzou^♦︎

^* Equal contribution.

^●Virginia Tech^♦︎PLAN Lab · University of Illinois Urbana-Champaign

CVPR 2026

Paper arXiv Code (coming soon)Dataset (coming soon)

TL;DR

We introduce 3D-VCD, a training-free method for reducing hallucinations in embodied agents reasoning about 3D scenes. By contrasting predictions from the observed scene with deliberately distorted versions, 3D-VCD suppresses answers unsupported by the scene’s objects and geometry.

3D-VCD contrasts predictions from observed and distorted 3D scene graphs.

Hallucination mitigation in 3D scenes

3D-VCD contrasts predictions from observed and distorted scene graphs to suppress unsupported answers without retraining.

Abstract

Large multimodal models are increasingly used as the reasoning core of embodied agents operating in 3D environments, yet they remain prone to hallucinations that can produce unsafe and ungrounded decisions. Existing inference-time hallucination mitigation methods largely target 2D vision-language settings and do not transfer to embodied 3D reasoning, where failures arise from object presence, spatial layout, and geometric grounding rather than pixel-level inconsistencies. We introduce 3D-VCD, the first inference-time visual contrastive decoding framework for hallucination mitigation in 3D embodied agents. 3D-VCD constructs a distorted 3D scene graph by applying semantic and geometric perturbations to object-centric representations, such as category substitutions and coordinate or extent corruption. By contrasting predictions under the original and distorted 3D contexts, our method suppresses tokens that are insensitive to grounded scene evidence and are therefore likely driven by language priors. We evaluate 3D-VCD on the 3D-POPE and HEAL benchmarks and show that it consistently improves grounded reasoning without any retraining, establishing inference-time contrastive decoding over structured 3D representations as an effective and practical route to more reliable embodied intelligence.

Contributions

3D-VCD as a New Inference Framework. We introduce 3D-VCD, the first training-free, inference-time contrastive decoding framework for hallucination mitigation in 3D embodied agents.
Semantic and Geometric Distortion Operators. We propose a simple and effective 3D counterfactual grounding mechanism that constructs distorted scene graphs through semantic and geometric perturbations, and uses dual-context logit fusion to suppress predictions unsupported by the underlying 3D evidence.
Strong Generalization Without Retraining. We demonstrate 3D-VCD improves grounded reasoning across embodied hallucination benchmarks, reducing over-affirmation on 3D-POPE and lowering hallucination on HEAL with minimal computational overhead.

3D-VCD Method

Overview of our Method. Given 3D environment observations, 3D-VCD constructs a structured 3D scene graph (𝒢) that encodes object categories, centroids, and extents. It then applies controlled semantic and geometric perturbations to produce a distorted version of the environment (𝒢^{^}). Given a textual query (x), the MLLM agent processes both contexts in parallel. 3D-VCD then contrastively fuses the resulting logits to identify and suppress hallucination-prone tokens. This training-free procedure enforces 3D-grounded reasoning at inference time.

Quantitative Results

3D-POPE comparison

Results on the 3D-POPE Benchmark. Our 3D-VCD model achieves the highest precision, accuracy, and F1-score across all evaluation categories (Random, Popular, and Adversarial), outperforming prior 3D language models such as 3D-LLM, 3D-VisTA, and LEO. Notably, it significantly reduces Yes-rate (e.g., 99.81% → 75.15% in the Random set) while consistently improving precision and accuracy. These results demonstrate that 3D-VCD effectively mitigates over-affirmation bias and hallucination, yielding more balanced and reliable predictions in 3D reasoning.

3D-POPE comparison
Model	Training-free	Precision ↑	Recall ↑	F1 ↑	Accuracy ↑	Yes (%) ↓
Random
Random Baseline	—	50.00	50.00	50.00	50.00	50.00
3D-LLM [13]	No	50.03	99.88	66.67	50.07	99.81
3D-VisTA [48]	No	50.12	53.58	51.79	49.66	53.95
LEO	No	51.95	77.65	62.25	52.91	74.73
3D-VCD (Ours)	Yes	62.16	92.90	74.48	67.99	75.15
Popular
Random Baseline	—	50.00	50.00	50.00	50.00	50.00
3D-LLM [13]	No	49.97	99.88	66.61	49.94	99.94
3D-VisTA [48]	No	47.40	51.88	49.54	49.49	52.30
LEO	No	48.30	77.65	59.55	47.27	80.38
3D-VCD (Ours)	Yes	52.35	92.86	66.95	54.00	89.02
Adversarial
Random Baseline	—	50.00	50.00	50.00	50.00	50.00
3D-LLM [13]	No	49.97	99.88	66.61	49.94	99.94
3D-VisTA [48]	No	48.28	54.39	51.15	51.14	52.99
LEO	No	48.47	77.98	59.78	47.52	80.45
3D-VCD (Ours)	Yes	52.90	92.59	67.32	54.92	87.82

HEAL: distractor injection

Object and state hallucination rates when prompts introduce distracting objects. Lower CHAIR scores are better; VCD rows use the corresponding Llama or Qwen backbone.

HEAL: distractor injection
Model	Object hallucination · CO (%) ↓	State hallucination · CS (%) ↓
Llama-3-8B-Instruct-VCD	2.39	12.43
Llama-3-8B-Instruct	2.58	9.49
Qwen-14B-Instruct-VCD	3.55	5.00
Qwen-14B-Instruct	4.13	16.45
Gemma-2-9b-it	11.60	5.70
DS-R1-Distil-LLaMA-8B	13.90	27.20

CO and CS denote object and state hallucination rates. Best scores are bold; second-best scores are underlined.

Ablation Studies

Effect of Scene Graph Distortion Types. We conduct a systematic study to analyze how different scene graph perturbations affect the performance of 3D- VCD. By independently corrupting semantic, geometric, and structural attributes, we isolate how each type of information contributes to grounded reasoning.

Robustness across graph perturbations

AdversarialPopularRandom

Key Insight:
3D- VCD consistently improves F1 from 0.63 → 0.74–0.77 across all distortion types.

Semantic Corruptions. We perturb object categories through synonym substitution and modifier removal to test robustness to lexical ambiguity and reduced contextual specificity.

Low-SemSub High-SemSub SemDropMod

Geometric Corruptions. We introduce Gaussian noise to object centroids and extents to simulate sensor noise and imperfect 3D reconstruction.

Low-Geom High-Geom

Structural Corruptions. We modify scene structure through object sparsification, relation flipping, and distractor injection to test reliance on relational and contextual cues.

Struct-RelFlip Struct-Dist Struct-Sparse

Mixed Corruptions. We combine semantic and geometric perturbations to mimic realistic embodied sensing noise where category uncertainty and spatial variance co-occur.

Semantic + Geometric

Across all settings, 3D- VCD consistently outperforms the baseline, demonstrating strong robustness to corrupted 3D inputs. These results indicate that contrastive decoding effectively leverages discrepancies between clean and perturbed scene representations, reinforcing grounded reasoning and suppressing hallucinations.

Qualitative Results

We present qualitative examples showing how 3D-VCD improves factual grounding across both 3D-POPE and HEAL. Across these examples, the baseline model exhibits several failure modes, including missing objects that are present, hallucinating objects that do not exist, and generating spurious ungrounded content. In contrast, 3D-VCD predictions remain more faithful to the scene graph and the underlying physical environment.

Taken together, these examples highlight the dual advantage of 3D-VCD: it reduces false positives by suppressing hallucinated objects and attributes, while also improving recall by attending to grounded, semantically coherent evidence. This leads to more reliable 3D reasoning in both embodied question answering and action-oriented grounding settings.

Qualitative comparison on 3D-POPE: Scene0424_00

The baseline incorrectly predicts the absence of a dining table, missing the true table object in the scene. In contrast, 3D-VCD correctly grounds and identifies the dining table by aligning contrastive decoding with the scene graph.

HEAL Example: brushing_lint_off_clothing

The baseline Qwen-14B-Instruct model hallucinates a nonexistent microwave.n.01_1 in its symbolic goal prediction. In contrast, 3D-VCD produces clean symbolic goals with no hallucinated objects, correctly grounding all sweaters on the bed and removing dust-related states as required by the instruction. The right panel shows the first-person view of the agent interacting with the sweaters in the scene.

Qualitative comparison on 3D-POPE: Scene0608_00

The baseline 3D-LLM hallucinates the presence of a bed. In contrast, 3D-VCD correctly answers No by contrasting logits under perturbed 3D scene graphs, effectively suppressing hallucinated object activations. The right panel shows the object-level scene segmentation for reference.

Qualitative comparison on 3D-POPE: Scene0479_00

The baseline hallucinates a desk object is present, referring instead to a counter in the scene. In contrast, 3D-VCD correctly determines that no desk exists by suppressing spurious category matches through contrastive decoding aligned with the object-centric scene graph.

BibTeX

@inproceedings{ogunleye20263dvcd,
  title={3D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decoding},
  author={Ogunleye, Makanjuola Adekunmi and Abdelrahman, Eman and Lourentzou, Ismini},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2026}
}

Template acknowledgements

This site is built upon the work of Nerfies, made available under the Creative Commons Attribution-ShareAlike 4.0 International License. We gratefully acknowledge 3D-GRAND, and HEAL for open-sourcing their models, data, and code.