Online Adaptation for Implicit Object Tracking and
Shape Reconstruction in the Wild

RA-L 2022, IROS 2022

1UC San Diego, 2TuSimple

We introduce a unified framework for joint object tracking and shape reconstruction
which leverages the continuity in video data.


Tracking and reconstructing 3D objects from cluttered scenes are the key components for computer vision, robotics and autonomous driving systems. While recent progress in implicit function has shown encouraging results on high-quality 3D shape reconstruction, it is still very challenging to generalize to cluttered and partially observable LiDAR data.

In this paper, we propose to leverage the continuity in video data. We introduce a novel and unified framework which utilizes a neural implicit function to simultaneously track and reconstruct 3D objects in the wild. Our approach adapts the DeepSDF model (i.e., an instantiation of the implicit function) in the video online, iteratively improving the shape reconstruction while in return improving the tracking, and vice versa. We experiment with both Waymo and KITTI datasets and show significant improvements over state-of-the-art methods for both tracking and shape reconstruction tasks.



Overall Framework

After initialization of the shape code, tracking and shape adaptation are performed iteratively. At a specific frame, the incoming object point cloud is first aligned to the previous shape, and then the shape is adapted to the aligned point cloud. Both procedures are based on DeepSDF.


Iterative optimization

We optimize pose by pushing the point cloud to the zero-level set of the SDF field in (a), and deform the SDF field to match the point cloud in (b) for better shape. We use different temperature of the color to represent the distances from the surface.

Optimization (a) Object tracking
Optimization (b) Shape adaptation


Thanks to the proposed online adaptation mechanism, the performance of both tracking and reconstruction can be improved. In the following, we show the tracking results as well as the shape evolution process.


  author    = {Ye, Jianglong and Chen, Yuntao and Wang, Naiyan and Wang, Xiaolong},
  title     = {Online Adaptation for Implicit Object Tracking and Shape Reconstruction in the Wild},
  journal   = {IEEE Robotics and Automation Letters},
  year      = {2022},