Lecture 12 : Principal Components

Consider the standard setting where we are given n points in d dimensions. Call these ~ x1, ~ x2, . . . , ~ xn. As before, our goal is to reduce the number of dimensions to a small number k. In principal component analysis (or PCA), we will model the data by a k-dimensional subspace, and find the subspace for which the error in this representation is smallest. Suppose k = 1. Then we want to approximate the data with a line. Assume the data is centered, so that ∑n i=1 ~ xi = 0, and that the line passes through the origin. Let the line correspond to direction ~ w, a unit vector. What is the error in approximating ~ xi with ~ w? We can use the perpendicular distance between the point ~ xi and the line represented by ~ w. It is easy to check that the perpendicular is given by ~ xi − (~ xi · ~ w)~ w, so that its squared length is