Compressive Epsilon Photography for Post-Capture Control in Digital Imaging

Abstract

compressive-epsilon-photography-post-capture-control-computational-imaging-lab-iit-madras

Description

Method

  1. Per-pixel intensity analysis. As focus and aperture vary, each pixel's intensity traces a smooth, unimodal profile ➯ peaking at the setting where that pixel is sharply in focus. This low-frequency, highly structured behaviour is what makes the whole stack compressible.
  2. GMM learning. Per-pixel intensity profiles from training data are clustered with k-means (100–450 clusters), and a PCA basis is learned around each cluster center, capturing the redundancy across the dataset.
  3. Greedy sampling. Starting from the learned GMM, we greedily choose the m focus-aperture settings to actually capture by picking, at each step, the setting that most reduces the MMSE reconstruction error over all unobserved settings, updating the posterior covariance after every selection.
  4. Geometric & photometric calibration. Changing focus and aperture subtly warps and re-vignettes the image, so before reconstruction we calibrate for magnification, lens distortion and translation (geometric) and for vignetting relative to reference images (photometric), to sub-pixel accuracy.
  5. Stack reconstruction. For every focus-aperture setting that wasn't captured, we compute the posterior GMM parameters and use an MMSE estimator to infer the per-pixel intensity, combining it with the calibrated photometric parameters to render the full image.

Results

compressive-epsilon-photography-post-capture-control-computational-imaging-lab-iit-madras

Figure 1: Compressive epsilon photography enables post-capture freedom from a few carefully selected photographs of a scene. We enable complete post-capture freedom by estimating the entire collection of photographs corresponding to varying camera parameters from a few select photographs at pre-determined settings.

compressive-epsilon-photography-greedy-image-selection-computational-imaging-lab-iit-madras

Figure 5. Greedy image selection ➯ PSNR vs. number of acquired images. Our greedy sampling scheme gives almost 10 dB improvement over traditional sampling.

compressive-epsilon-photography-animal-dataset-focal-stack-computational-imaging-lab-iit-madras

Figure 7. Focal stack reconstruction of the “Animal” dataset ➯ a 45-image stack reconstructed from just 8 captures, at 25.8 dB.

compressive-epsilon-photography-tennis-dataset-reconstruction-computational-imaging-lab-iit-madras

Figure 14. Focus-aperture stack reconstruction for an outdoor “tennis” scene ➯ 24 settings reconstructed from 16 input images.

Visualizations

compressive-epsilon-photography-project-summary-computational-imaging-lab-iit-madras

Summary: Compressive epsilon photography at a glance ➯ capturing 8–16 images and computationally reconstructing hundreds of focus/aperture settings while maintaining full spatial resolution.

compressive-epsilon-photography-focus-aperture-stack-reconstruction-computational-imaging-lab-iit-madras

Figure. Focus-aperture stack reconstruction on the Tennis Racquet dataset ➯ 16 captured input images (highlighted) are used to reconstruct all 810 focus×aperture combinations.

compressive-epsilon-photography-per-pixel-intensity-profiles-computational-imaging-lab-iit-madras

Figure. The key insight: per-pixel intensity varies smoothly and unimodally with focus setting, which is why a GMM can capture 99.6% of the signal energy with only a few hundred components.

compressive-epsilon-photography-greedy-sampling-strategy-computational-imaging-lab-iit-madras

Figure. Greedy sampling strategy ➯ the proposed selection consistently beats uniform and random sampling, giving roughly a 3–9 dB PSNR improvement across image budgets from 2 to 32.

compressive-epsilon-photography-experimental-validation-computational-imaging-lab-iit-madras

Figure. Experimental validation ➯ 793→16 image reduction for confocal stereo, 25–32 dB reconstruction SNR across datasets, 99.6% GMM energy captured, and a 13-subject human study where 66% found reconstructions acceptable or better.

compressive-epsilon-photography-detailed-results-computational-imaging-lab-iit-madras

Figure. Detailed results across five datasets (Chess, Animals, Liquid, Glass Ball, Tennis) ➯ reconstruction SNR stays above the 25 dB quality threshold everywhere, alongside the human perception study and key contributions.

compressive-epsilon-photography-traditional-vs-compressive-animation-computational-imaging-lab-iit-madras

Animation: Traditional epsilon photography (hundreds to thousands of captures, hours of acquisition) vs. the compressive approach (a handful of optimally-selected images, seconds to capture, full post-capture control).

compressive-epsilon-photography-focus-sweep-animation-computational-imaging-lab-iit-madras

Animation. Focus sweep across all 45 reconstructed focal planes, computed entirely from the sparse input capture.

compressive-epsilon-photography-aperture-depth-of-field-animation-computational-imaging-lab-iit-madras

Animation. Depth-of-field variation as the reconstructed aperture is swept from F/2.2 to F/16.

compressive-epsilon-photography-reconstruction-quality-vs-input-images-animation-computational-imaging-lab-iit-madras

Animation. Reconstruction PSNR climbing as more input images are captured, from 4 images up to the full 16-image budget.

Applications

  • Casual photography: adjust focus and aperture after the shot is taken, so there's no need to get every setting right at the moment of capture.
  • Computational microscopy: recover a 3D reconstruction from a 2D focal stack while minimizing acquisition time and, with it, sample degradation.
  • Confocal stereo depth estimation: traditional confocal stereo needs 793 images per scene; the paper shows competitive depth estimates from as few as 8 compressively captured images, an ~99% reduction in acquisition.
  • Extended depth of field: compose professional-grade all-in-focus images from the reconstructed stack, at quality superior to a single all-focus capture.

Advantages & Limitations

  • No hardware modifications ➯ works with any standard digital camera.
  • Full spatial resolution is retained, unlike light-field cameras.
  • Burst-mode capture ➯ the 8–16 input images can be acquired in seconds.
  • Per-pixel processing preserves fine detail and edges without neighborhood assumptions.
  • Greedy sampling gives roughly a 10 dB improvement over uniform/random sampling.
  • Generalizes across diverse scene types ➯ chess pieces, animals, liquids, glass, and outdoor scenes were all tested.

Publications

Acknowledgments