Enhancing Glass Surface Reconstruction via Depth Prior for Robot Navigation

Abstract

Indoor robot navigation is often compromised by glass surfaces, which severely corrupt depth sensor measurements. While foundation models like Depth Anything 3 provide excellent geometric priors, they lack an absolute metric scale.

We propose a training-free framework that leverages depth foundation models as a structural prior, employing a robust local RANSAC-based alignment to fuse it with raw sensor depth. This naturally avoids contamination from erroneous glass measurements and recovers an accurate metric scale. Furthermore, we introduce GlassRecon, a novel RGB-D dataset with geometrically derived ground truth for glass regions. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art baselines, especially under severe sensor depth corruption. The dataset and related code are available at github.com/jarvisyjw/GlassRecon.

We assume most glass structures in indoor environments — windows, glass doors, tabletops — are planar. Images are sourced from Matterport3D, which provides RGB-D data and camera intrinsics, and RGB-D GSD, which contains images with manually labeled glass surfaces selected from Matterport3D.

Based on the RGB-D data and glass masks, we construct the dataset through a multi-step processing pipeline that generates complete depth maps for glass regions. The raw sensor depth map fails to measure glass surfaces accurately, so further annotation is required.

The computed depths replace the original values within the glass mask. Where a glass region lacks any depth point annotations, that area is masked and excluded from evaluation. To ensure reliability, each corrected depth map is converted to a 3D point cloud and manually inspected; only images where the glass plane appears geometrically consistent are retained.

The hard subset contains samples with larger glass regions exhibiting significant sensor error. Each image is accompanied by an RGB image, raw sensor depth map, binary glass mask, annotated ground-truth depth, and camera intrinsics.

GlassRecon/
├── images/                  # RGB images (PNG)
├── intrinsics/              # JSON files with camera intrinsics & depth scale
├── masks/                   # Binary masks for glass regions (PNG)
├── sensor_depths/           # Raw sensor depth maps (PNG)
├── completed_depths/        # Completed depth maps (NPY)
└── evaluation_depths/
    ├── depths_npy/          # Filtered depth maps (NPY) – glass regions
    │                        #   that could not be completed are masked out
    ├── depths_vis/          # Visualizations (PNG)
    └── pointclouds/         # 3D point clouds (PLY) – back-projected from
                             #   depths_npy using intrinsics

To demonstrate practical effectiveness, we evaluate the impact of our depth correction on mapping and navigation through two complementary experiments: one on the public ScanNet++ dataset, and one using real-world data collected in campus environments.

Evaluation on ScanNet++. Using the nvblox reconstruction framework on a scene with a glass balcony door, the map generated from raw sensor depth exhibits severe holes around the glass, rendering the area unsafe for navigation. Our corrected depth recovers the planar structure of the glass surface, yielding a complete and geometrically consistent map.

Real-world validation. We conducted field experiments in the lobbies of two campus buildings, recording two RGB-D sequences with an iPhone 17 Pro in areas featuring multiple glass doors. Consistent with the ScanNet++ findings, raw sensor depth treats glass regions as empty voids, whereas our corrected depth reconstructs these areas with complete, accurate geometry.

Map fidelity is especially important for planning methods whose candidate edges are validated through collision checking. To demonstrate the impact on navigation, we converted the reconstructed point cloud of Scene02 into a 2D occupancy grid for path planning. When using the map from raw sensor depth, the planner misinterprets glass doors as traversable open space, resulting in a collision-prone path. Our corrected depth accurately registers glass surfaces as physical obstacles, enabling a safe, collision-free route.

If you find our work useful, please cite:

@article{zheng2026enhancing,
  title={Enhancing Glass Surface Reconstruction via Depth Prior for Robot Navigation},
  author={Zheng, Jiamin and Yu, Jingwen and Chen, Guangcheng and Zhang, Hong},
  journal={arXiv preprint arXiv:2604.18336},
  year={2026}
}