Computer Vision for Light-Field Medical Imaging: FPGA Image Processing System and Depth Acquisition GAN

Abstract

Light field imaging demands a higher data bandwidth and more post-processing than traditional photographic imaging, a characteristic that is compounded in real-time medical applications. Rendering a realistic sub-aperture image captured from a light field array requires a precise 3D understanding of the scene being imaged. For use in a surgical environment, these images must be captured and rendered with consistently high resolution and low latency. In this paper, we present two systems to improve the latency and precision of the Proprio Vision medical light field array. The first of these improvements is a hardware and software system that implements an FPGA as a modular platform to pre-process and intelligently downsample images sent from the image sensors before they are processed by the main PC in the array, reducing the workload of the main GPU. The second proposed system is a neural network that generates depth maps from light field images. This system reduces depth capture time and rendering latency by replacing the current structured-light depth capture solution. The model is implemented as a Generative Adversarial Network to improve the accuracy and realism of the generated depth maps. Together, the two proposed systems reduce the hardware cost and complexity of the Proprio Vision array while improving latency and image quality.