Optimal HDR and Depth from Dual Cameras

Abstract

hdr-and-depth-from-dual-cameras-computational-imaging-lab-iit-madras
  • Generalizes Hasinoff et al.'s noise-optimal single-camera HDR framework to dual-camera capture, jointly optimizing for HDR quality and disparity accuracy.
  • Formulates exposure/ISO selection as a capture-time-minimizing optimization subject to coverage, disparity and SNR constraints, and shows the disparity constraint makes the problem non-convex ➯ addressed with a dedicated initialization scheme.
  • Releases the first stereo HDR dataset with dense exposure-ISO stacks from a smartphone dual camera: 6 scenes, ~142 images per scene.

Description

Problem Statement

Coverage
The entire radiance range of the scene must be captured without saturation in at least one camera.
Disparity
The fraction of pixels with erroneous disparity must stay below a chosen threshold.
SNR
The per-exposure signal-to-noise ratio must exceed a minimum threshold.
Time
Total capture time across both cameras is minimized subject to the above.

Method

Log radiance interval framework

log(x_l) − log(t) + log(g)  <  log(φ)  <  log(x_u) − log(t) + log(g)

Optimization formulation

minimize    t_cap                              (total capture time)

subject to  coverage:   entire radiance range captured in ≥1 camera
            disparity:  overlap constraint between camera pairs
            SNR:        per-exposure SNR ≥ η, for every image

Processing pipeline

1. Scene radiance analysis

Extract the radiance distribution (histogram) of the input scene to understand its dynamic range and content statistics.

2. Optimal sequence identification

Solve the optimization problem above to compute the exposure and ISO settings that balance capture time, SNR, and disparity requirements.

3. Stereo image capture

Simultaneously capture images from both cameras using the optimized exposure-ISO sequence.

4. Iterative refinement

Alternate between disparity map estimation and inverse camera response function (ICRF) estimation over multiple iterations until convergence.

5. HDR fusion

Combine the aligned exposures using the estimated ICRFs and disparity information to produce the final HDR radiance map.

6. Output generation

Produce the final HDR image and depth map, ready for downstream applications such as refocusing and view synthesis.

Visual Explanations

optimal-hdr-depth-dual-cameras-optimization-convergence-animation-computational-imaging-lab-iit-madras

Optimization in progress. Candidate exposure placements for the primary (P1–P3) and secondary (S1–S3) cameras are adjusted against the scene's radiance distribution as capture time, disparity error, SNR and coverage are jointly evaluated during initialization.

optimal-hdr-depth-dual-cameras-log-radiance-interval-concept-computational-imaging-lab-iit-madras

Log radiance interval concept. Each exposure on the primary and secondary camera covers an interval of the scene's log radiance; the shaded overlap between cameras is what makes stereo matching (and hence disparity estimation) possible.

optimal-hdr-depth-dual-cameras-processing-pipeline-architecture-computational-imaging-lab-iit-madras

Processing pipeline. From scene radiance analysis through optimal sequence selection, disparity/ICRF refinement, and HDR fusion, to the final HDR image and depth map.

optimal-hdr-depth-dual-cameras-project-overview-diagram-computational-imaging-lab-iit-madras

Project overview. Input scene radiance distribution flows through the optimization framework to produce the optimal capture sequence and, ultimately, the HDR + depth output.

A one-page quick-reference summary of the problem, solution, dataset and metrics below is also available: view quick-reference sheet.

Results

optimal-hdr-depth-dual-cameras-main-qualitative-comparison-computational-imaging-lab-iit-madras

Figure 8: Comparison of HDR and depth reconstructions for various capture schemes ➯ our optimal exposure/ISO sequence recovers both HDR radiance and depth more accurately.

optimal-hdr-depth-dual-cameras-quantitative-figure-computational-imaging-lab-iit-madras

Figure 5. Performance of the proposed method for HDR-only capture, compared across exposure/ISO strategies.

optimal-hdr-depth-dual-cameras-method-application-figure-computational-imaging-lab-iit-madras

Figure 12. Results of refocusing on HDR reconstructions ➯ desirable results are obtained on both test scenes.

Quantitative Results

Table 1 from the paper: average comparison for the HDR-only recovery task, "3Shot" capture vs. our optimal sequence, relative to the single-camera full-stack ground truth.
Metric3Shot (baseline)Ours (optimal sequence)Full stack (ground truth)
HDR‑VDP‑261.4368.55—
Capture time (s)0.3000.2060.319
Speedup over full stack1.05×1.42×1×

Mathematical Foundation

Signal-to-noise ratio model

SNR(φ, t_i, g_i)  =  φ² t_i  /  (2φ t_i + σ_r² + σ_q² g_i²)

Non-convexity of the disparity constraint

Stereo HDR Dataset

  • Camera: LG G5 smartphone dual-camera system.
  • Scenes: 6 diverse indoor and outdoor scenes.
  • Images per scene: ~142 exposure-ISO sequences (~852 images total).
  • Coverage: full exposure and ISO stacks captured for each scene, as stereo pairs.
  • Exposure range: 3.2s to 1/4000s (12+ stops).
  • ISO range: 50 to 400+.
  • Ground truth: disparity maps and depth estimates.
  • Release: available upon publication.

Applications

Related Work

HDR imaging. Multi-image HDR is well established, with prior work on exposure sequence selection, camera response function estimation, and multi-image fusion; more recent deep learning approaches show promising results for both static and dynamic scenes.

Optimal capture sequences. Hasinoff et al.'s noise-optimal framework is the theoretical starting point for this work, establishing the SNR advantages of controlling ISO alongside exposure time. We extend these ideas to the stereo setting, where the additional disparity requirement adds real complexity to the optimization.

Stereo HDR. Earlier work in this space has addressed registration, exposure-invariant matching, and motion compensation between stereo captures. To our knowledge, this is the first work to optimize the capture sequence itself for joint HDR and depth recovery, rather than only the post-capture processing.

Depth estimation. We build on established stereo matching algorithms (Mozerov et al.) for disparity estimation; our contribution is in providing an optimal input capture sequence for these algorithms to work from, rather than in the matching algorithm itself.

Future Directions

Video HDR

Extend the framework to temporal sequences and moving scenes, incorporating optical flow and motion compensation.

Deep learning integration

Combine the optimization framework with neural networks for end-to-end HDR and depth estimation.

Multi-camera systems

Generalize the formulation to systems with three or more cameras for improved coverage and robustness.

Real-time implementation

Develop efficient algorithms for on-device computation on smartphones and embedded systems.

Adaptive algorithms

Create scene-adaptive strategies that adjust capture parameters in real time based on scene content.

BibTeX

@article{chari2020optimal,
  title   = {Optimal HDR and Depth from Dual Cameras},
  author  = {Chari, Pradyumna and Vadathya, Anil Kumar and Mitra, Kaushik},
  journal = {arXiv preprint arXiv:2003.05907},
  year    = {2020}
}

Publications