Brief Tutorial Description:

Signal denoising is closely related to function estimation from noisy samples. The same problem is also addressed in statistics (nonlinear regression) and neural network learning. Vapnik-Chervonenkis (VC) theory has recently emerged as a general theory for estimation of dependencies from finite samples. This theory emphasizes model complexity control according to Structural Risk Minimization (SRM) inductive principle, which considers a nested set of models of increasing complexity (called a structure), and then selects an optimal model complexity providing minimum error for future samples. This tutorial shows how to apply the framework of VC-theory to signal estimation / denoising. There are 3 factors important for accurate signal estimation from finite samples: (1) the type of (orthogonal) basis functions used (i.e., Fourier basis, wavelets etc) (2) the choice of a structure, i.e., ordering of the basis functions according to their 'importance' for accurate signal estimation. This corresponds to the choice of a 'structure' under SRM formulation. (3) selecting an optimal number of terms (basis functions) from the ordered sequence of basis functions (2), aka model selection or complexity control (in statistics). We describe methodology for specifying appropriate orderings (2) and an analytic expression for model selection (3) for signal processing applications. We also present empirical comparisons between the proposed methodology and current state-of-the-art wavelet thresholding methods for univariate signals. These comparisons suggest that the prudent choice of a structure (2) and the use of VC-based model selection (3) are critical for accurate signal estimation with finite samples.

This tutorial consists of 3 parts: (1) Background on VC learning theory. We provide a brief introduction to VC-theory as it relates to signal/image denoising applications (2) VC-based framework for signal denoising. We describe VC-based methodology for signal denoising, contrast it to other recent denoising methods (such as wavelet thresholding) and show some empirical comparisons using synthetic univariate signals. (3) Applications. We present several empirical studies using real-life data. Applications include: ECG signal denoising, denoising of functional MRI data and 2D image denoising.