Optimal HDR and Depth from Dual Cameras
Pradyumna Chari, Anil Kumar Vadathya, Kaushik Mitra · UCLA, Rice University, IIT Madras · arXiv, 2020
Abstract
- Generalizes Hasinoff et al.'s noise-optimal single-camera HDR framework to dual-camera capture, jointly optimizing for HDR quality and disparity accuracy.
- Formulates exposure/ISO selection as a capture-time-minimizing optimization subject to coverage, disparity and SNR constraints, and shows the disparity constraint makes the problem non-convex ➯ addressed with a dedicated initialization scheme.
- Releases the first stereo HDR dataset with dense exposure-ISO stacks from a smartphone dual camera: 6 scenes, ~142 images per scene.
Description
Problem Statement
- Coverage
- The entire radiance range of the scene must be captured without saturation in at least one camera.
- Disparity
- The fraction of pixels with erroneous disparity must stay below a chosen threshold.
- SNR
- The per-exposure signal-to-noise ratio must exceed a minimum threshold.
- Time
- Total capture time across both cameras is minimized subject to the above.
Method
Log radiance interval framework
log(x_l) − log(t) + log(g) < log(φ) < log(x_u) − log(t) + log(g)
Optimization formulation
minimize t_cap (total capture time)
subject to coverage: entire radiance range captured in ≥1 camera
disparity: overlap constraint between camera pairs
SNR: per-exposure SNR ≥ η, for every image
Processing pipeline
1. Scene radiance analysis
Extract the radiance distribution (histogram) of the input scene to understand its dynamic range and content statistics.
2. Optimal sequence identification
Solve the optimization problem above to compute the exposure and ISO settings that balance capture time, SNR, and disparity requirements.
3. Stereo image capture
Simultaneously capture images from both cameras using the optimized exposure-ISO sequence.
4. Iterative refinement
Alternate between disparity map estimation and inverse camera response function (ICRF) estimation over multiple iterations until convergence.
5. HDR fusion
Combine the aligned exposures using the estimated ICRFs and disparity information to produce the final HDR radiance map.
6. Output generation
Produce the final HDR image and depth map, ready for downstream applications such as refocusing and view synthesis.
Visual Explanations
Optimization in progress. Candidate exposure placements for the primary (P1–P3) and secondary (S1–S3) cameras are adjusted against the scene's radiance distribution as capture time, disparity error, SNR and coverage are jointly evaluated during initialization.
Log radiance interval concept. Each exposure on the primary and secondary camera covers an interval of the scene's log radiance; the shaded overlap between cameras is what makes stereo matching (and hence disparity estimation) possible.
Processing pipeline. From scene radiance analysis through optimal sequence selection, disparity/ICRF refinement, and HDR fusion, to the final HDR image and depth map.
Project overview. Input scene radiance distribution flows through the optimization framework to produce the optimal capture sequence and, ultimately, the HDR + depth output.
A one-page quick-reference summary of the problem, solution, dataset and metrics below is also available: view quick-reference sheet.
Results
Figure 8: Comparison of HDR and depth reconstructions for various capture schemes ➯ our optimal exposure/ISO sequence recovers both HDR radiance and depth more accurately.
Figure 5. Performance of the proposed method for HDR-only capture, compared across exposure/ISO strategies.
Figure 12. Results of refocusing on HDR reconstructions ➯ desirable results are obtained on both test scenes.
Quantitative Results
| Metric | 3Shot (baseline) | Ours (optimal sequence) | Full stack (ground truth) |
|---|---|---|---|
| HDR‑VDP‑2 | 61.43 | 68.55 | — |
| Capture time (s) | 0.300 | 0.206 | 0.319 |
| Speedup over full stack | 1.05× | 1.42× | 1× |
Mathematical Foundation
Signal-to-noise ratio model
SNR(φ, t_i, g_i) = φ² t_i / (2φ t_i + σ_r² + σ_q² g_i²)
Non-convexity of the disparity constraint
Stereo HDR Dataset
- Camera: LG G5 smartphone dual-camera system.
- Scenes: 6 diverse indoor and outdoor scenes.
- Images per scene: ~142 exposure-ISO sequences (~852 images total).
- Coverage: full exposure and ISO stacks captured for each scene, as stereo pairs.
- Exposure range: 3.2s to 1/4000s (12+ stops).
- ISO range: 50 to 400+.
- Ground truth: disparity maps and depth estimates.
- Release: available upon publication.
Applications
Related Work
HDR imaging. Multi-image HDR is well established, with prior work on exposure sequence selection, camera response function estimation, and multi-image fusion; more recent deep learning approaches show promising results for both static and dynamic scenes.
Optimal capture sequences. Hasinoff et al.'s noise-optimal framework is the theoretical starting point for this work, establishing the SNR advantages of controlling ISO alongside exposure time. We extend these ideas to the stereo setting, where the additional disparity requirement adds real complexity to the optimization.
Stereo HDR. Earlier work in this space has addressed registration, exposure-invariant matching, and motion compensation between stereo captures. To our knowledge, this is the first work to optimize the capture sequence itself for joint HDR and depth recovery, rather than only the post-capture processing.
Depth estimation. We build on established stereo matching algorithms (Mozerov et al.) for disparity estimation; our contribution is in providing an optimal input capture sequence for these algorithms to work from, rather than in the matching algorithm itself.
Future Directions
Video HDR
Extend the framework to temporal sequences and moving scenes, incorporating optical flow and motion compensation.
Deep learning integration
Combine the optimization framework with neural networks for end-to-end HDR and depth estimation.
Multi-camera systems
Generalize the formulation to systems with three or more cameras for improved coverage and robustness.
Real-time implementation
Develop efficient algorithms for on-device computation on smartphones and embedded systems.
Adaptive algorithms
Create scene-adaptive strategies that adjust capture parameters in real time based on scene content.
BibTeX
@article{chari2020optimal,
title = {Optimal HDR and Depth from Dual Cameras},
author = {Chari, Pradyumna and Vadathya, Anil Kumar and Mitra, Kaushik},
journal = {arXiv preprint arXiv:2003.05907},
year = {2020}
}