Measuring 3D-Length from an Image Using Robust and Accurate Curve Estimation

Abstract

Accurate and robust curve estimation is important for many practical applications. In this work we develop an application for measuring an infant’s 3D length with regular 2D cellphone photos using accurate and robust curve estimation. Measuring 3D length of a deformable object from a 2D RGB image is a very challenging task. The 2D length that appears in the 2D image could represent very different actual 3D lengths depending on the distance of the object to the camera and the camera pose. The depth information of the object parts is lost when the object is projected from the 3D world to the 2D image. Furthermore, a deformable and dynamic object such as an infant moves all the time, which could introduce severe motion blur in the 2D image. The infant’s skin is relatively textureless and lacks easily identifiable feature points. There is no existing reliable method that could estimate its physical length directly from a 2D image. 3D pose estimation based on learning usually does not infer 3D length with enough accuracy. To reliably and accurately calculate the 3D length from a 2D image, we propose a new technique which uses easily obtainable and easy-to-apply circular stickers with a known size to help create feature points and provide reference physical lengths at the 3D locations of the feature points. A circular sticker in the 3D space will be projected into an ellipse in the 2D image. The sticker’s 3D location could be recovered with high accuracy from a single RGB image if its ellipse projection in the 2D image could be precisely estimated. With the help of the ellipse estimation, the length of a deformable object such as an infant could be calculated with decent accuracy. However, estimating the exact location of an elliptical curve from real world images, which contain various kind of noise and motion blur, is not a trivial task. Few existing ellipse estimation methods can reliably detect the ellipses and accurately estimate the ellipse parameters to obtain the 3D lengths with enough accuracy to meet our requirement (for infant length measurement). To solve the problem, we develop a deformable contour based method called ellipse growing for reliably detecting the curve of the ellipse from the images and estimate the ellipse parameters accurately. For images with strong motion blur, we could deblur the images before the ellipse parameter estimation. However, existing deblurring algorithms may introduce artifacts that make ellipse parameter estimation even harder and less accurate. Instead, we propose a CNN-based method to help restore the ellipses. It is fast and able to handle typical motion blur of real-world images. Neural networks require a lot of training data. We propose an approach to automatically generate enough realistic training data for training the neural networks to obtain accurate results. The proposed approach can be generalized for estimating other shapes in the images. Inspired by the CNN’s effectiveness, we further generalize the CNN to recover more general smooth curves from motion blurred and noisy images without restoration. We developed novel loss functions, new differentiable layers, and synthetic data generation methods. Our model is light weighted, and is very effective on real-world photos. The overall pipeline could be easily used for more general curve estimation tasks.