0

I wrote a R script which import data from fcs files, perform a clustering by FlowSOM and I'd like to check it and visualize my data on t-SNE. It works well for low amount of data but when I try with a lot of data (around 4 thousand cells) it crashes when it reaches 310 000 cells without any error message.

out_tsne <- Rtsne(data_Rtsne, 
              perplexity = perp,
              initial_dims = 50,
              max_iter = i, 
              pca=T,
              verbose = T,
              num_threads=0)

data_Rtsne is a matrix of 4000000*10.

With a smaller dataset (around 500 000 cells), it works without trouble. So it should be a memory limit issue.

Can someone explain to me why I don't have any error message from R (it closes and that's all) and why can't I see a saturation of RAM on windows' task manager?

Thank in advance for your help!

EDIT : Ok, I tried again with a smaller dataset (around 500 000 cells) and it works without trouble. So it should be a memory limit issue. Can someone explain to me why I don't have any error message from R (it closes and that's all) and why can't I see a saturation of RAM on windows' task manager ?

Florian M.
  • 19
  • 3
  • 1
    Does this happen only when running the function in Rstudio? Do you monitor RAM usage by any chance? – Roman Luštrik Jan 17 '22 at 17:35
  • No, it happens too when running in R. Yes, I followed it. It didn't pass over 10% of it ... – Florian M. Jan 17 '22 at 19:50
  • This is likely a memory limit issue. – Phil Jan 17 '22 at 21:10
  • Do you think ? Without any error message ? I tried once to change it with memory.limit(size=xxx), according to my 128GO of RAM but it still crashes :( – Florian M. Jan 17 '22 at 21:47
  • If memory seems fine then there's other low hanging fruit things you could try, like trying out a different R version. You can use Docker to pull prepared images, fire up a container with linking your data in using the `-v` flag and running the script. Sorry if I'm explaining things you already know. I recommend using [rocker](https://github.com/rocker-org/rocker) R Docker images. – Roman Luštrik Jan 19 '22 at 06:26
  • Thank you for all of this. I don't know rocker and usage of Docker. I'll take a look on it. Thank you for your advices ! (I'm just an immunologist's PhD student who likes to use R but I learn by myself and it's the first time with "big data") – Florian M. Jan 19 '22 at 08:11

0 Answers0