0

I got 2 matrices each with 200k records (one is a large get_sentences - review_phrases, the other is review_scores). Binded them in to a data frame and need to write it on a csv but get a memory error. What should i do? Do the packages bigmemory or ff help?

I'm using Windows 10 64bit 8GB RAM. memory limits give the following output:

memory.limit()
[1] 1.759219e+13
memory.size()
[1] 418.85

Here's the code:

  sentiment_matrix_final = data.frame (Comment = review_phrases)

 df_scores = data.frame(Scores = review_scores)

 sentiment_matrix_final = cbind(sentiment_matrix_final, df_scores)

 sentiment_matrix_final = data.frame (Comment = review_phrases, Score = review_scores)

 str(sentiment_matrix_final)


  # Write CSV in R
  write.csv(sentiment_matrix_final, file = "Score_sentiment_test.csv")

I get the output:

>sentiment_matrix_final = data.frame (Comment = review_phrases)

> df_scores = data.frame(Scores = review_scores)

> sentiment_matrix_final = cbind(sentiment_matrix_final, df_scores)
Error: cannot allocate vector of size 750 Kb
The_Coder
  • 29
  • 5
  • you can try : ``memory.limit(yourRAMsize_inMb + x)``. The ``x`` part will allow R to use your disk if the RAM is not enough. So let say you have 32GB ram and want to allow R 16GB of disk you could try ``memory.limit(49152)``. You should not rely on this but sometimes it can help. – Gainz Jun 27 '19 at 14:38
  • Mine memory.limits returns `memory.limit() [1] 1.759219e+13` Is it enough? – The_Coder Jun 27 '19 at 14:50
  • I'm not sure this make sens, that would mean you got 17592190000000 Mb. How much RAM do you have when you look in your system info? – Gainz Jun 27 '19 at 14:53
  • I got 64bit with 8GB RAM – The_Coder Jun 27 '19 at 15:00
  • Ok, then can you try ``memory.limit(24000)`` and run your code please? If it doesn't work then maybe your code need to be optimize. – Gainz Jun 27 '19 at 15:01
  • Tried it: `Warning message: In memory.limit(24000) : cannot decrease memory limit: ignored` – The_Coder Jun 27 '19 at 15:09
  • Seems like R think you really have "``memory.limit() [1] 1.759219e+13``". It's strange because the maximum of ``memory.limit()`` is 8Tb RAM for 64 bits Windows in R. – Gainz Jun 27 '19 at 15:13
  • I'm sorry I can't help much without your datas. Your code seems fine so I am not sure what the problem is. – Gainz Jun 27 '19 at 15:15
  • Basically you want to write a CSV with two columns that corresponds to your two vectors. You can write by batchs appending to your file. – F. Privé Jun 27 '19 at 15:26
  • This isn't reproducible - 750 kb is nothing. You should look at the task manager to see what is taking up all the memory. Separately, instead of using ```cbind()```, just try the 4th and last lines of your first block of code. – Cole Jun 27 '19 at 16:29

0 Answers0