Math Blog #4: Regularization from the spectral point of view

In Math blog #2, I presented regularization as the method for overcoming the difficulties with solving ill-posed problems of the form z=Au. We mentioned that the problem could be made well-posed by solving instead

(1) u_alpha = argmin_u {||Au-z||^2 + alpha*||u||^2},

which is also the solution of the 'normal equations' for (1); that is,

(A'*A+alpha*I)u_alpha = A'*z,

where I is the identity operator (or matrix in the discrete case) and alpha>0. This is known as Tikhonov regularization.

Tikhonov regularization can also be viewed spectrally. Recall from Math blog #3 that the inverse of A has a spectral representation via the singular value expansion (SVE) of A: 

(2) Ainv*z = \sum_i {V_i*<U_i,z>/s_i}.

The problem with (2) for ill-posed problems, as we've discussed previously, is that {1/s_i} is unbounded (or has extremely large maximum in the discrete case). 

Tikhonov regularization (1) can be equivalently expressed in terms of the SVE of A as follows:

(3) u_alpha = \sum_i {V_i*<U_i,z>/s_i^alpha}, 

where s_i^\alpha=(s_i^2+alpha)/s_i. Note that if alpha=0, (3) and (2) are the same, but for alpha>0, {1/s_i^alpha} is now bounded (or has a much smaller maximum than {1/s_i} in the discrete case), making z-->u_alpha a stable (or more stable in the discrete case) mapping.