Connecting to Spark with Sparklyr gives Permission Denied Error

Question

After installing sparklyr package I followed the instruction here ( http://spark.rstudio.com/ ) to connect to spark. But faced with this error. Am I doing something wrong. Please help me.

 sc = spark_connect( master = 'local' )

Error in file(con, "r") : cannot open the connection In addition: Warning message: In file(con, "r") : cannot open file 'C:\Users\USER\AppData\Local\Temp\RtmpYb3dq4\fileff47b3411ae_spark.log': Permission denied

But I am able to find the file at the stated location. And on opening, I found it to be empty.I

did you load it: library(sparklyr)? – MLavoie Jun 26 '17 at 09:24 — MLavoie, Jun 26 '17 at 09:24

score 1 · Answer 1 · answered Jun 27 '17 at 11:41

1

First of all, did you install sparklyr from github devtools::install_github("rstudio/sparklyr") or CRAN? There were some issues some time ago with Windows installations.

The issue you have seems to be related to TEMP and TMP folder level permission on Windows or to file creation permission. Every time you start sc <- spark_connect(), it tries to create a folder and file to write the log files. Make sure you have a write access to these locations.

answered Jun 27 '17 at 11:41

michalrudko

1,432
2
16
30

I have installed it from CRAN. Ok checking for write access. – boral Jun 28 '17 at 19:24
if write access was not a cause, I'd suggest to uninstall the package and install from github – michalrudko Jun 29 '17 at 07:32
I was one version backdated in R. So have updated R as well as updated sparklyr from github. The issue is solved then. – boral Jul 01 '17 at 20:13

user1767316 · Answer 2 · 2019-12-01T12:07:03.963

I could observe the same error message with version 2.4.3 and 2.4.4 in different cases:

When trying to connect to a non "local" master, using spark_connect(master="spark://192.168.0.12:7077", ..),

if the master is not started or not responding at the specified master url.
when setting a specific incomplete configuration
- in my case trying to set dynamicAllocation to true, without other required dynamicAllocation settings:

conf <- spark_config() conf$spark.dynamicAllocation.enabled <- "true"

Connecting to Spark with Sparklyr gives Permission Denied Error

2 Answers2