Under-Display Camera (UDC) technology is key to true full-screen smartphone experiences while keeping the front-facing camera commonly used for selfies. But placing a camera below the period pattern of the display panel introduces significant diffraction effects and image quality challenges. A primary artifact we encounter is a highly degraded Point Spread Function (PSF), often appearing as a diffractive star pattern around bright light sources due to the display's pixel structure. This PSF manifests visibly as a pronounced halo effect around high-contrast edges severely impacting image quality and resolution in these boundary regions.
The conventional Image Signal Processor (ISP) pipeline, relying on sequential, decoupled stages (demosaicing, denoising, sharpening), is limited in inverting this complex, signal-dependent degradation. Traditional deconvolution methods struggle because the UDC PSF has high dynamic range and so is highly challenging to deconvolve without artifacts in real-world applications. Existing phone makers resort to blurring or smoothing out boundary regions to avoid such artifacts resulting in loss of detail.
At Glass Imaging, we tackle this problem head-on using our Neural ISP instead to recover detail. Our approach is a holistic, end-to-end restoration network, trained explicitly on the forward model of the entire UDC imaging system—from the display to the sensor. By learning the inverse function of the UDC-specific degradation, the Neural ISP simultaneously performs demosaicing, denoising, and complex deconvolution in a single, highly optimized pass. This neural approach allows us to effectively mitigate the severe PSF and diffraction artifacts and maintain detail and textures.
We tested this idea using the front facing under-display camera on a Xiaomi Mix 4 smartphone. As demonstrated in the comparison below, the image processed by our Neural ISP shows dramatically reduced halo-ing and a restoration of sharp, high-fidelity edges, fundamentally surpassing the capabilities of standard UDC processing solutions. This is a crucial step toward achieving parity between UDC and traditional front-facing camera performance.
Glass Imaging is paving the way for under-display cameras to become a standard feature in future smartphones by using our deep learning pipeline to correct optical aberrations. While our initial efforts have focused on enhancing still photography, we recognize the growing importance of front-facing video. Consequently, Glass has been hard at work on a new Neural ISP pipeline specifically for smartphone video—keep an eye out for our upcoming update on under-display camera video performance!
