Asymptotics of Gaussian Regularized Least Squares
We consider regularized least-squares (RLS) with a Gaussian kernel. We prove that if we let the Gaussian bandwidth σ → ∞ while letting the regularization parameter λ → 0, the RLS solution tends to a polynomial whose order is controlled by the rielative rates of decay of 1/ σ1 and λ: if λ = σ-(2k+1), then, as σ → ∞, the RLS solution tends to the kth order polynomial with minimal empirical error. We illustrate the result with an example.