Feature Tracking for VSLAM
Most algorithms for visual SLAM (simultaneous localization and mapping) track feature points in the scene, in the form of small planar surface patches. However, tracking can fail if the features are not planar. This happens frequently in outdoor scenes, for example when tracking points on vegetation. It also happens frequently in indoor scenes, because feature points are often detected on occluding boundaries.
In this work, my graduate student and I developed a method to track feature patches that are partially planar; i.e., where only part of a patch is the projection of a planar surface. We developed an algorithm to segment each image patch to identify the pixels that belong to the dominant plane, and determine the three dimensional structure of that plane. The algorithm improves the accuracy of VSLAM, especially in scenes where surface discontinuities are common.
For full details of this work, see our journal article: A. Masoud and W. Hoff, "Segmentation and tracking of nonplanar templates to improve VSLAM," Robotics and Autonomous Systems, vol. 86, pp. 29-56, 2016 (pdf).
In the videos below, a RED square is a newly 2D feature, where we have just begun to track it. The square turns GREEN when the feature has been tracked sufficiently long, that we can estimate its 3D location. Finally, a BLUE dot indicates that the point has been used in a bundle adjustment process.
This is the “usual” method of tracking planar patch features, for VSLAM. Note how features frequently are lost and new features must be acquired.
RED: Tracked points in 2D only. GREEN: Tracked points in 3D. BLUE DOT: Point is used in bundle adjustment.
To avoid the problem of frequently dropping features, our method incrementally learns which pixels belong to the “dominant" plane, and estimates the probability of each pixel as belonging to the plane. It also estimates the parameters of the dominant plane. This allows us to more accurately predict the appearance of the patch in a new image, which improves our ability to find it and track it.
The figure to the left shows examples of tracked features from this scene. The top row shows the location of each patch; the middle row are the actual image patches, and the bottom row is the probability of each pixel as belonging to a plane.
This is our method, where we detect and track non-planar patch features. Note how features are much more likely to be tracked over a longer period of time.
RED: Tracked points in 2D only. GREEN: Tracked points in 3D. BLUE DOT: Point is used in bundle adjustment.
This video shows a 3D plot of the reconstructed point cloud and the camera positions.
The trajectory estimated using our method of tracking non-planar features (called the “Partial method”) closely tracks the ground truth. The trajectory estimated using the typical method of tracking planar features (called the “Whole method”) quickly diverges from the ground truth.
