How is the complexity of PCA O(min(p^3,n^3))?

Question

I've been reading a paper on Sparse PCA, which is: http://stats.stanford.edu/~imj/WEBLIST/AsYetUnpub/sparse.pdf

And it states that, if you have n data points, each represented with p features, then, the complexity of PCA is O(min(p^3,n^3)).

Can someone please explain how/why?

Don Reba · Accepted Answer · 2013-12-11T00:33:57.217

43

Covariance matrix computation is O(p²n); its eigen-value decomposition is O(p³). So, the complexity of PCA is O(p²n+p³).

O(min(p³,n³)) would imply that you could analyze a two-dimensional dataset of any size in fixed time, which is patently false.

edited Dec 11 '13 at 00:33

answered Dec 11 '13 at 00:05

Don Reba

13,814
3
48
61

1

It's odd how the paper phrases this vaguely as "involves a search for directions." It does not outright say that this is the algorithm's complexity, just strongly implies it. – Don Reba Dec 11 '13 at 00:14
Great! Could you please give a reference for the above so that it would be easier to cite? – Ébe Isaac Jan 15 '18 at 06:30
@ÉbeIsaac Covariance matrix complexity follows immediately from definition. There are lower-complexity algorithms for eigenvalue decomposition, but they are close to O(p³), and this is probably the complexity the paper's author assumes. You shouldn't cite SO answers as authoritative sources, though, unless they are from Jon Skeet. – Don Reba Feb 04 '18 at 06:42

michaelt · Answer 2 · 2018-03-10T14:46:59.187

Assuming your dataset is $X \in \R^{nxp}$ where n: number of samples, d: dimensions of a sample, you are interested in the eigenanalysis of $X^TX$ which is the main computational cost of PCA. Now matrices $X^TX \in \R^{pxp}$ and $XX^T \in \R^{nxn}$ have the same min(n, p) non negative eigenvalues and eigenvectors. Assuming p less than n you can solve the eigenanalysis in $O(p^3)$. If p greater than n (for example in computer vision in many cases the dimensionality of sample -number of pixels- is greater than the number of samples available) you can perform eigenanalysis in $O(n^3)$ time. In any case you can get the eigenvectors of one matrix from the eigenvalues and eigenvectors of the other matrix and do that in $O(min(p, n)^3)$ time.

$$X^TX = V \Lambda V^T$$

$$XX^T = U \Lambda U^T$$

$$U = XV\Lambda^{-1/2}$$

Unfortunately there is no latex support, I suggest you use backquotes to format that as code, or export your latex fomulas to png and upload that. — Ash, Mar 10 '18 at 14:51

score 2 · Answer 3 · edited Sep 24 '19 at 08:41

Below is michaelt's answer provided in both the original LaTeX and rendered as a PNG.

Image of LaTeX answer

LaTeX code:

Assuming your dataset is $X \in R^{n\times p}$ where n: number of samples, p: dimensions of a sample, you are interested in the eigenanalysis of $X^TX$ which is the main computational cost of PCA. Now matrices $X^TX \in \R^{p \times p}$ and $XX^T \in \R^{n\times n}$ have the same min(n, p) non negative eigenvalues and eigenvectors. Assuming p less than n you can solve the eigenanalysis in $O(p^3)$. If p greater than n (for example in computer vision in many cases the dimensionality of sample -number of pixels- is greater than the number of samples available) you can perform eigenanalysis in $O(n^3)$ time. In any case you can get the eigenvectors of one matrix from the eigenvalues and eigenvectors of the other matrix and do that in $O(min(p, n)^3)$ time.

please post any code as actual code, images are not helpful – Azsgy Mar 30 '18 at 20:38 — Azsgy, Mar 30 '18 at 20:38

How is the complexity of PCA O(min(p^3,n^3))?

3 Answers3

Linked