Compressive Epsilon Photography for Post-Capture Control in Digital Imaging
(ACM SIGGRAPH / ACM Transactions on Graphics, 2014)
Abstract
Description
Method
- Per-pixel intensity analysis. As focus and aperture vary, each pixel's intensity traces a smooth, unimodal profile ➯ peaking at the setting where that pixel is sharply in focus. This low-frequency, highly structured behaviour is what makes the whole stack compressible.
- GMM learning. Per-pixel intensity profiles from training data are clustered with k-means (100–450 clusters), and a PCA basis is learned around each cluster center, capturing the redundancy across the dataset.
- Greedy sampling. Starting from the learned GMM, we greedily choose the m focus-aperture settings to actually capture by picking, at each step, the setting that most reduces the MMSE reconstruction error over all unobserved settings, updating the posterior covariance after every selection.
- Geometric & photometric calibration. Changing focus and aperture subtly warps and re-vignettes the image, so before reconstruction we calibrate for magnification, lens distortion and translation (geometric) and for vignetting relative to reference images (photometric), to sub-pixel accuracy.
- Stack reconstruction. For every focus-aperture setting that wasn't captured, we compute the posterior GMM parameters and use an MMSE estimator to infer the per-pixel intensity, combining it with the calibrated photometric parameters to render the full image.
Results
Figure 1: Compressive epsilon photography enables post-capture freedom from a few carefully selected photographs of a scene. We enable complete post-capture freedom by estimating the entire collection of photographs corresponding to varying camera parameters from a few select photographs at pre-determined settings.
Figure 5. Greedy image selection ➯ PSNR vs. number of acquired images. Our greedy sampling scheme gives almost 10 dB improvement over traditional sampling.
Figure 7. Focal stack reconstruction of the “Animal” dataset ➯ a 45-image stack reconstructed from just 8 captures, at 25.8 dB.
Figure 14. Focus-aperture stack reconstruction for an outdoor “tennis” scene ➯ 24 settings reconstructed from 16 input images.
Visualizations
Summary: Compressive epsilon photography at a glance ➯ capturing 8–16 images and computationally reconstructing hundreds of focus/aperture settings while maintaining full spatial resolution.
Figure. Focus-aperture stack reconstruction on the Tennis Racquet dataset ➯ 16 captured input images (highlighted) are used to reconstruct all 810 focus×aperture combinations.
Figure. The key insight: per-pixel intensity varies smoothly and unimodally with focus setting, which is why a GMM can capture 99.6% of the signal energy with only a few hundred components.
Figure. Greedy sampling strategy ➯ the proposed selection consistently beats uniform and random sampling, giving roughly a 3–9 dB PSNR improvement across image budgets from 2 to 32.
Figure. Experimental validation ➯ 793→16 image reduction for confocal stereo, 25–32 dB reconstruction SNR across datasets, 99.6% GMM energy captured, and a 13-subject human study where 66% found reconstructions acceptable or better.
Figure. Detailed results across five datasets (Chess, Animals, Liquid, Glass Ball, Tennis) ➯ reconstruction SNR stays above the 25 dB quality threshold everywhere, alongside the human perception study and key contributions.
Animation: Traditional epsilon photography (hundreds to thousands of captures, hours of acquisition) vs. the compressive approach (a handful of optimally-selected images, seconds to capture, full post-capture control).
Animation. Focus sweep across all 45 reconstructed focal planes, computed entirely from the sparse input capture.
Animation. Depth-of-field variation as the reconstructed aperture is swept from F/2.2 to F/16.
Animation. Reconstruction PSNR climbing as more input images are captured, from 4 images up to the full 16-image budget.
Applications
- Casual photography: adjust focus and aperture after the shot is taken, so there's no need to get every setting right at the moment of capture.
- Computational microscopy: recover a 3D reconstruction from a 2D focal stack while minimizing acquisition time and, with it, sample degradation.
- Confocal stereo depth estimation: traditional confocal stereo needs 793 images per scene; the paper shows competitive depth estimates from as few as 8 compressively captured images, an ~99% reduction in acquisition.
- Extended depth of field: compose professional-grade all-in-focus images from the reconstructed stack, at quality superior to a single all-focus capture.
Advantages & Limitations
- No hardware modifications ➯ works with any standard digital camera.
- Full spatial resolution is retained, unlike light-field cameras.
- Burst-mode capture ➯ the 8–16 input images can be acquired in seconds.
- Per-pixel processing preserves fine detail and edges without neighborhood assumptions.
- Greedy sampling gives roughly a 10 dB improvement over uniform/random sampling.
- Generalizes across diverse scene types ➯ chess pieces, animals, liquids, glass, and outdoor scenes were all tested.
