0

I try to perform example from https://cran.r-project.org/web/packages/text2vec/vignettes/files-multicore.html but with my file "text" - 3.7Gb plain text, build from Wikipedia XML dump with Perl script from here - http://mattmahoney.net/dc/textdata.html 

setwd("c:/rtest")
library(text2vec)
library(doParallel)
N_WORKERS = 2
registerDoParallel(N_WORKERS)
it_files_par = ifiles_parallel(file_paths = "text")
it_token_par = itoken_parallel(it_files_par, preprocessor = tolower, tokenizer = word_tokenizer)
vocab = create_vocabulary(it_token_par)

This causes error:

Error in unserialize(socklist[[n]]) : error reading from connection

I have 8Gb RAM, word2vec model from this file is created without any errors.

семен антонов
  • 17
  • 5

1 Answers1

0

First of all it doesn't make sense to use parallel iterators on a single file - each file processed in a separate R worker process. So here it will be worse than just itoken. Also it involves sending result from each worker to the master process. Here we see that result it too big to be send through socket. Long story short - just use itoken or split your file into several smaller files.

Dmitriy Selivanov
  • 4,545
  • 1
  • 22
  • 38