Robust RVM Regression Using Sparse Outlier Model
(IEEE CVPR 2010)
Abstract
Key Contributions
●Novel Noise Model
Explicit decomposition of noise into Gaussian and sparse outlier components enables principled outlier handling.
●Computational Efficiency
The robust RVM problem is reduced to a larger standard RVM problem, letting it leverage fast existing solvers.
●Dual Approaches
Both a Bayesian (probabilistic) and an optimization-based method are provided, with complementary strengths.
●Strong Empirical Results
Outperforms standard RVM and prior robust methods on both real and synthetic datasets.
Problem Statement
Why Is Robustness Important?
- Sensor noise and measurement errors
- Data entry mistakes
- Distribution shifts and anomalies
- Corrupted or mislabeled samples
Limitations of Existing Methods
| Method | Approach | Limitation |
|---|---|---|
| Standard RVM | Assumes Gaussian noise | Highly sensitive to outliers; unreliable estimates |
| Mixture of Gaussians | Multi-component noise model | Complex inference; requires expensive variational methods |
| Student's t-distribution | Heavy-tailed noise model | Elegant but computationally expensive |
| Trimmed Likelihood | Iterative subset selection | Sensitive to initial estimates; slow convergence |
This Work: Proposes efficient robust RVM methods that maintain computational tractability while handling outliers explicitly.
Technical Approach
Model Formulation
Standard RVM
y = Φw + e
Proposed Decomposition
y = Φw + n + s
Approach 1: Robust Bayesian RVM (RB-RVM)
Joint Sparse Prior
- Model parameters: p(w|α) = ∏ N(w_i | 0, α_i-1)
- Outliers: p(s|β) = ∏ N(s_i | 0, β_i-1)
Inference via Evidence Maximization
- Maximize marginal likelihood p(y|α,β,σ²)
- Use EM algorithm (or fast RVM solver)
- Obtain point estimates: α*, β*, σ*²
Posterior Prediction
- Compute posterior p(w,s|y,α*,β*,σ*²)
- Use posterior mean for predictions
- Posterior variance gives uncertainty estimates
Approach 2: Basis Pursuit Denoising (BP-Denoising)
Optimization Formulation
Minimize ||w_s||0 subject to ||y − Ψw_s||22 ≤ ε
Convex Relaxation
Replace L0 norm with L1 norm for tractability:
minimize ||w_s||1 subject to ||y − Ψw_s||22 ≤ ε
Solution Method
- Iterative algorithms with convergence guarantees
- Sparse solutions naturally emerge
- Complementary to the Bayesian approach
Description
Results
Figure 6: Salt and pepper noise removal experiment ➯ RB-RVM effectively removes sparse outliers while preserving fine image details, outperforming conventional RVM and the median filter.
Figure 1. Prediction by RVM, RB-RVM, and BP-RVM for the symmetric outliers case, with detected outliers marked.
Figure 3. Prediction error vs. outlier fraction for the symmetric and asymmetric outlier cases.
Figure 11. Mean absolute error of age prediction vs. fraction of controlled outliers on the FG-NET dataset.
Experimental Validation
Experiment 1: Image Denoising
Experiment 2: Age Estimation from Faces
| Method | Inlier MAE | Outlier MAE | All MAE |
| Standard RVM | N/A (no inlier/outlier split) | N/A | 6.80 |
| Robust Bayesian RVM | 4.61 | 25.87 | 6.52 |
Advantages Over Prior Work
vs. Mixture of Gaussians Model
● Simpler inference (no expensive variational methods) ● Direct outlier estimation without mixture complexity
vs. Student's t-distribution
● Computationally more efficient ● Explicit separation of inlier and outlier noise components
vs. Trimmed Likelihood
● No need for robust initial subset estimation ● Joint optimization of parameters and outliers ● Faster convergence (fewer iterations)
Computational Advantage: The robust RVM problem reduces to a standard RVM on an augmented feature space, enabling use of existing fast RVM solvers.
Applications
Computer Vision
- Age estimation from facial images
- Head pose and facial feature estimation
- 3D human pose from images or video
- Lighting and material estimation
- Gaze direction prediction
Image Processing
- Image denoising and restoration
- Super-resolution with corrupted inputs
- Medical image reconstruction (MRI, CT)
- Artifact removal in medical imaging
- Image inpainting with missing data
Machine Learning & Data Science
- Robust feature learning with noisy labels
- Time series prediction with anomalies
- Domain adaptation with distribution shift
- Outlier detection in regression problems
- Anomaly identification in datasets
Visualizations
Animation: Step-by-step walkthrough of the RB-RVM pipeline ➯ observing the noisy data, decomposing the noise, setting priors, and fitting the model with outlier detection.
Animation: Fitting quality comparison across Standard RVM, Robust Bayesian RVM, and Basis Pursuit Denoising as outliers are progressively identified.
Summary Figure. RB-RVM and BP-Denoising processing flows (top), alongside the underlying noise decomposition concept and RMSE vs. outlier contamination (bottom) ➯ robust methods stay flat as Standard RVM's error grows with contamination.
Key Insights
1. Explicit Noise Decomposition Works: By explicitly separating Gaussian and outlier noise, we achieve better interpretability and robustness than implicit heavy-tailed distributions.
2. Computational Efficiency Maintained: Reformulating as a larger RVM preserves computational tractability ➯ a key advantage over prior variational methods.
3. Dual Methods Are Complementary: Bayesian RVM provides uncertainty quantification; BP-Denoising provides principled optimization ➯ use together for best results.
4. Real-World Impact: On the full FG-NET age-estimation dataset, RB-RVM automatically separates inliers from outliers (MAE 4.61 vs. 25.87 years) and its inlier accuracy is close to the contemporary state-of-the-art, demonstrating practical value beyond synthetic benchmarks.
