Robust RVM Regression Using Sparse Outlier Model

Abstract

robust-rvm-regression-sparse-outlier-model-computational-imaging-lab-iit-madras

Key Contributions

●Novel Noise Model

Explicit decomposition of noise into Gaussian and sparse outlier components enables principled outlier handling.

●Computational Efficiency

The robust RVM problem is reduced to a larger standard RVM problem, letting it leverage fast existing solvers.

●Dual Approaches

Both a Bayesian (probabilistic) and an optimization-based method are provided, with complementary strengths.

●Strong Empirical Results

Outperforms standard RVM and prior robust methods on both real and synthetic datasets.

Problem Statement

Why Is Robustness Important?

  • Sensor noise and measurement errors
  • Data entry mistakes
  • Distribution shifts and anomalies
  • Corrupted or mislabeled samples

Limitations of Existing Methods

MethodApproachLimitation
Standard RVMAssumes Gaussian noiseHighly sensitive to outliers; unreliable estimates
Mixture of GaussiansMulti-component noise modelComplex inference; requires expensive variational methods
Student's t-distributionHeavy-tailed noise modelElegant but computationally expensive
Trimmed LikelihoodIterative subset selectionSensitive to initial estimates; slow convergence

This Work: Proposes efficient robust RVM methods that maintain computational tractability while handling outliers explicitly.

Technical Approach

Model Formulation

Standard RVM

y = Φw + e

Proposed Decomposition

y = Φw + n + s

Approach 1: Robust Bayesian RVM (RB-RVM)

Joint Sparse Prior
  • Model parameters: p(w|α) = ∏ N(w_i | 0, α_i-1)
  • Outliers: p(s|β) = ∏ N(s_i | 0, β_i-1)
Inference via Evidence Maximization
  1. Maximize marginal likelihood p(y|α,β,σ²)
  2. Use EM algorithm (or fast RVM solver)
  3. Obtain point estimates: α*, β*, σ*²
Posterior Prediction
  • Compute posterior p(w,s|y,α*,β*,σ*²)
  • Use posterior mean for predictions
  • Posterior variance gives uncertainty estimates

Approach 2: Basis Pursuit Denoising (BP-Denoising)

Optimization Formulation

Minimize ||w_s||0 subject to ||y − Ψw_s||22 ≤ ε

Convex Relaxation

Replace L0 norm with L1 norm for tractability:

minimize ||w_s||1 subject to ||y − Ψw_s||22 ≤ ε

Solution Method
  • Iterative algorithms with convergence guarantees
  • Sparse solutions naturally emerge
  • Complementary to the Bayesian approach

Description

Results

robust-rvm-regression-sparse-outlier-model-main-result-computational-imaging-lab-iit-madras

Figure 6: Salt and pepper noise removal experiment ➯ RB-RVM effectively removes sparse outliers while preserving fine image details, outperforming conventional RVM and the median filter.

robust-rvm-regression-method-computational-imaging-lab-iit-madras

Figure 1. Prediction by RVM, RB-RVM, and BP-RVM for the symmetric outliers case, with detected outliers marked.

robust-rvm-regression-quantitative-computational-imaging-lab-iit-madras

Figure 3. Prediction error vs. outlier fraction for the symmetric and asymmetric outlier cases.

robust-rvm-regression-application-computational-imaging-lab-iit-madras

Figure 11. Mean absolute error of age prediction vs. fraction of controlled outliers on the FG-NET dataset.

Experimental Validation

Experiment 1: Image Denoising

Experiment 2: Age Estimation from Faces

Method Inlier MAE Outlier MAE All MAE
Standard RVMN/A (no inlier/outlier split)N/A6.80
Robust Bayesian RVM4.6125.876.52

Advantages Over Prior Work

vs. Mixture of Gaussians Model

● Simpler inference (no expensive variational methods)    ● Direct outlier estimation without mixture complexity

vs. Student's t-distribution

● Computationally more efficient    ● Explicit separation of inlier and outlier noise components

vs. Trimmed Likelihood

● No need for robust initial subset estimation    ● Joint optimization of parameters and outliers    ● Faster convergence (fewer iterations)

Computational Advantage: The robust RVM problem reduces to a standard RVM on an augmented feature space, enabling use of existing fast RVM solvers.

Applications

Computer Vision

  • Age estimation from facial images
  • Head pose and facial feature estimation
  • 3D human pose from images or video
  • Lighting and material estimation
  • Gaze direction prediction

Image Processing

  • Image denoising and restoration
  • Super-resolution with corrupted inputs
  • Medical image reconstruction (MRI, CT)
  • Artifact removal in medical imaging
  • Image inpainting with missing data

Machine Learning & Data Science

  • Robust feature learning with noisy labels
  • Time series prediction with anomalies
  • Domain adaptation with distribution shift
  • Outlier detection in regression problems
  • Anomaly identification in datasets

Visualizations

robust-rvm-regression-algorithm-animation-computational-imaging-lab-iit-madras

Animation: Step-by-step walkthrough of the RB-RVM pipeline ➯ observing the noisy data, decomposing the noise, setting priors, and fitting the model with outlier detection.

robust-rvm-regression-performance-comparison-animation-computational-imaging-lab-iit-madras

Animation: Fitting quality comparison across Standard RVM, Robust Bayesian RVM, and Basis Pursuit Denoising as outliers are progressively identified.

robust-rvm-regression-flows-and-plots-computational-imaging-lab-iit-madras

Summary Figure. RB-RVM and BP-Denoising processing flows (top), alongside the underlying noise decomposition concept and RMSE vs. outlier contamination (bottom) ➯ robust methods stay flat as Standard RVM's error grows with contamination.

Key Insights

1. Explicit Noise Decomposition Works: By explicitly separating Gaussian and outlier noise, we achieve better interpretability and robustness than implicit heavy-tailed distributions.

2. Computational Efficiency Maintained: Reformulating as a larger RVM preserves computational tractability ➯ a key advantage over prior variational methods.

3. Dual Methods Are Complementary: Bayesian RVM provides uncertainty quantification; BP-Denoising provides principled optimization ➯ use together for best results.

4. Real-World Impact: On the full FG-NET age-estimation dataset, RB-RVM automatically separates inliers from outliers (MAE 4.61 vs. 25.87 years) and its inlier accuracy is close to the contemporary state-of-the-art, demonstrating practical value beyond synthetic benchmarks.

Future Directions

Publications