Python: MemoryError for .corr(method='pearson').applymap

Question

I am trying to calculate the pearsons correlation for every single possible pair of two columns in a dataframe. I have 57997 columns. But I am getting a memory error.

t_logs = logs.T
print t_logs
results = t_logs.corr(method='pearson').applymap
print results[enter image description here][1]

Here is the trace back

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-99-d0010a131d17> in <module>()
       5 print logs
       6 
 ----> 7 results = t_logs.corr(method='pearson')
       8 print results

C:\Users\nne1s\Anaconda2\lib\site-packages\pandas\core\frame.pyc in 
corr(self, method, min_periods)
   4938 
   4939         if method == 'pearson':
-> 4940             correl = libalgos.nancorr(_ensure_float64(mat), 
minp=min_periods)
4941         elif method == 'spearman':
4942             correl = libalgos.nancorr_spearman(_ensure_float64(mat),

pandas\_libs\algos.pyx in pandas._libs.algos.nancorr 
(pandas\_libs\algos.c:15501)()

MemoryError:

picture of code linked here

Please don't post pictures of code or error messages. Post the full traceback here on SO instead. [The reason for this is explained here.](http://idownvotedbecau.se/imageofanexception/) — Mr. T, Jan 23 '18 at 01:22

score 0 · Answer 1 · answered Jan 23 '18 at 02:08

0

I don't think you want to calculate correlation coefficients for 57997 series.

If you're trying to get the correlation matrix for Log2 of Mean, Log2 of STD, etc. transpose the dataframe before you run corr

t_logs = t_logs.T
results = t_logs.corr(method='pearson')

I'm not sure what you wanted to do with applymap here

answered Jan 23 '18 at 02:08

usernamenotfound

1,540
2
11
18

can you post the entire traceback? – usernamenotfound Jan 23 '18 at 18:29
When I do it without transposing it I do not get the error. However, I need to to be transposed to take the correct correlation coefficient – Nneka Ede Jan 23 '18 at 18:29
what's the shape of the resulting dataframe when you use corr without transposing? – usernamenotfound Jan 23 '18 at 18:30
it is a 4x4. but essentially i need a 57997x57997. Maybe it is just too large – Nneka Ede Jan 23 '18 at 20:41
to help me find the relationship between 2 genes and their expression. – Nneka Ede Jan 23 '18 at 21:43

Python: MemoryError for .corr(method='pearson').applymap

1 Answers1