1

I have created an R package that makes use of the sparklyr capabilities within a dummy hello function. My package does a very simple thing as connection to a spark cluster, print the spark version and disconnect. The package is successfully clean and build and is successfully executed from R and Rstudio.

# Connect to Spark cluster
  spark_conn <-  sparklyr::spark_connect(master = "spark://elenipc.home:7077", spark_home = '/home/eleni/spark-2.2.0-bin-hadoop2.7/')

# Print the version of Spark
  sv<- sparklyr::spark_version(spark_conn)
  print(sv)

  # Disconnect from Spark
  sparklyr::spark_disconnect(spark_conn)

It is very important for me to be able to execute the hello function from OpenCpu rest api. (I have used opencpu api for executing many other custom created packages.) When invoking opencpu api like:

curl http://localhost/ocpu/user/rstudio/library/myFirstBigDataPackage/R/hello/print -X POST

i get the following response:

Failed while connecting to sparklyr to port (8880) for sessionid (89615): Gateway in port (8880) did not respond.
Path: /home/eleni/spark-2.2.0-bin-hadoop2.7/bin/spark-submit
Parameters: --class, sparklyr.Shell, '/home/rstudio/R/x86_64-pc-linux-gnu-library/3.4/sparklyr/java/sparklyr-2.2-2.11.jar', 8880, 89615
Log: /tmp/ocpu-temp/file26b165c92166_spark.log


---- Output Log ----
Error occurred during initialization of VM
Could not allocate metaspace: 1073741824 bytes

---- Error Log ----

In call:
force(code)

Of course allocate more memory to both java & spark executor does not resolve the issue. permission issues are also discarded as i already configured the etc/apparmor.d/opencpu.d/custom file so as to permit opencpu to have rwx privileges on spark. It seems to be a connectivity issue that i do not know how to face. During method invocation via opencpu api spark logs do not even print something.

For you info my environment configuration is as follows:

  • java version "1.8.0_65"
  • R version 3.4.1
  • RStudio version 1.0.153
  • spark-2.2.0-bin-hadoop2.7
  • opencpu 1.5 (compatible with my Ubuntu 14.04.3 LTS)

Thank you very much for you support and time!!!

Punit
  • 324
  • 1
  • 4
  • 17
efotopoulou
  • 139
  • 3
  • 13
  • Is there any way you can upgrade to a more recent version of OpenCPU? We fixed quite a bunch of issues in the 2.0 branch. – Jeroen Ooms Sep 28 '17 at 10:33
  • my linux version Ubuntu 14.04 (Trusty) does not support opencpu-2.0 that is supported by Ubuntu 16.04 (Xenial), but i will set up quickly a vm to test it – efotopoulou Sep 28 '17 at 11:16
  • @Jeroen I installed all from scratch to a fresh ubuntu vm 16.04 with opencpu-2.0. The message is the same with a more verbose output log. `---- Output Log ---- # # There is insufficient memory for the Java Runtime Environment to continue. # pthread_getattr_np # An error report file with more information is saved as: # /tmp/ocpu-temp/ocpu_session_688dfe5843/hs_err_pid1684.log [thread 140374552274688 also had an error]`` – efotopoulou Sep 28 '17 at 15:38
  • 2
    As of opencpu 2.0.4 the `/etc/opencpu/server.conf` file has a new option `enable.rlimits`. Can you try changing that to `false` and then `apachectl restart`? – Jeroen Ooms Sep 28 '17 at 21:17
  • @Jeroen Thanks a lot for your help!!! A fresh installation again resolved the issue!! opencpu is amazing and you very helpfull! – efotopoulou Sep 29 '17 at 14:45

0 Answers0