1

I recently start using SparkR. I have 1 master and 3 worker running and AWS. I am using RSudio Server. My codes involve stringr package, and this packages have been installed to /usr/share/R/library on all nodes.

But when I run SparkR:::map(data, function(x){str_replace(x, "XXXX", "XXxxx")}), i get the error 

could not find function "str_replace"

How can I load packages on worker nodes?

zero323
  • 322,348
  • 103
  • 959
  • 935
user2146141
  • 155
  • 1
  • 14

1 Answers1

3

To access function from an library you have either load and attach it:

library(stringr) # Or require(stringr)
str_replace(x, "XXXX", "XXxxx")

or use double colon operator:

stringr::str_replace(x, "XXXX", "XXxxx")

Unfortunately :: is quite expensive so if you prefer to keep your namespace clean you should consider creating a local binding:

str_replace <- stringr::str_replace
str_replace(x, "XXXX", "XXxxx")

On a side not using internal API with ::: is probably not the best idea. Ignoring good practices it is simply far to slow to be useful in practice.

zero323
  • 322,348
  • 103
  • 959
  • 935