Google’s nighttime photo experiments shed light on how smartphone cameras could get even better
Smartphone cameras are about to get even better.
Peyman Milanfar, a Google software engineer who had worked in the computational photography group, posted a lengthy analysis about using a smartphone camera to shoot nighttime photos with the same quality of an expensive DSLR.
Milanfar’s post chronicles his quest for high-quality nighttime images taken with a smartphone. DSLR cameras do well in this application, but smartphone cameras struggle. A DSLR can take good quality photos at night because it has a very large sensor that collects more light. The Nikon D500 DSLR boasts a 20.9 million pixel sensor with a pixel size of 4.2µm. The D500 sensor is enormous compared to the top ranked Google Pixel phone’s sensor with 12.3 million pixels that are 1.55µm. It also has a large, adjustable and precise lens that captures and focuses more light from the field of view on the sensor, reducing visual distortion, compared to the Pixel’s constrained, fixed camera lens assembly.
Capturing more light and better focus were the two smartphone camera problems Milanfar sought solve. It is hard to overcome the limitations of the Pixel and Nexus 6P phone hardware used in the experiment, but it is not impossible. This 51-second video explains how software can enhance limited light.
Milanfar solved the nighttime photo problem by writing a camera app that allowed him to manually control the camera settings:
- Exposure time: the length of time when the film or digital sensor is exposed to light
- ISO: measures the sensitivity of the image sensor
- Focus distance: the range of distance over which the camera can focus
Exposure time and ISO settings affect the amount of light. Milanfar set the exposure time to an unusually long time, 4 seconds, and set the ISO to a very sensitive 1600. A tripod was necessary to isolate the phone camera from handheld jitter. He set the focus distance to infinity.
He shot 32 burst images, then covered the lens and shot 32 more entirely black frames. The burst images were processed computing the mean values of all the pixels to remove the graininess. The black frames were used to remove faint grid patterns introduced by imperfections in the sensor.
The quality of the image of Point Reyes Lighthouse created with this process is very good.
For reference, the image below is the same scene shot with the phone held by hand using HDR+. The inset rectangle has been brightened in Photoshop to roughly match the previous picture.
Milanfar challenged himself with even more extreme shots through the experiment he chronicled, including nighttime stars on a clear dark night that left trails in the image because of the very long exposure time that he managed to remove with computational photography methods. If there is one conclusion, a talented software engineer can write software that enhances images to improve the quality images taken with a phone camera. It will never be possible to shoot the images used in the examples without a tripod, but machine learning can deliver much-improved photos taken with phones.
Applying machine learning to photos
Rapid and Accurate Image Super-Resolution (RAISR) was introduced last November by Google in a research paper. It uses a machine learning technique to produce high-quality versions of low-resolution images. The images are better than super resolution methods and are processed ten to a hundred times faster. RAISR is fast enough to run on a typical mobile device in real-time.
RAISR is a machine learning model, trained with low- and high-resolution images to learn to complete low-resolution images to look like high-resolution images. The image below shows how RAISR enhances the low-resolution image to look like a high-resolution image.
It is a process of adding missing pixels, not too different from a restoration artist’s repair of a work of art. The important point is that machine learning models can be trained to enhance a variety of low-quality images due to resolution, too little light because of an incorrect exposure time, or ISO setting or focus distance setting. The technology is still at a research paper stage; consequently, it may take a while before this technology works on consumer phones.
Machine learning models typically run on powerful GPU clusters have been adapted with a variety of techniques to run on smartphones. A good example is Facebook Applied Machine Learning group’s Camera Effects Platform, demonstrated earlier this month at the company’s F8 developer conference, that turns 2D images into 3D images using a machine learning technique called simultaneous localization and mapping (SLAM). Slam identifies planes and estimates volumetric space with just a smartphone camera. The 1-minute, 21-second video below demonstrates how a simple smartphone camera can discern depth in a field of view in roughly the same way a person would.
Research into understanding images by decomposing them to pixels dates back 30 years to Bell Labs. And research at NYU, the University of Toronto, the University of Montreal, Stanford and other top schools has been given much attention and big budgets by Amazon, Google and Facebook. Machine learning works.
Google photos, categorized by subject and image editing features, demonstrate this. Snapchat and Facebook’s theme transfer do, as well. Most of this runs on fast and sometimes specialized machines in the cloud, though now there is a race to distribute the models to run on phones, wearables and even Raspberry Pis. It is just a matter of time before the image-enhancing machine learning features reach smartphones to improve photo quality.
