0

I am trying to perform linear regression using SparklyR on an EMR cluster, and receiving the error below. The connection to Spark seems fine, and I have tried using several different datasets, but they all result in the same error. I am looking for insight as to what might be causing this error.

Code:

library(sparklyr)
library(dplyr)
Sys.setenv(SPARK_HOME="/usr/lib/spark")
config <- spark_config()
sc <- spark_connect(master = "yarn-client", config = config)

# copy mtcars into spark
mtcars_tbl <- copy_to(sc, mtcars)
# transform our data set, and then partition into 'training', 'test'
partitions <- mtcars_tbl %>%
  filter(hp >= 100) %>%
  mutate(cyl8 = cyl == 8) %>%
  sdf_partition(training = 0.5, test = 0.5, seed = 1099)

fit <- partitions$training %>%
  ml_linear_regression(response = "mpg", features = c("wt", "cyl"))

Error:

java.lang.IllegalArgumentException: invalid method setForceIndexLabel for object 88
    at sparklyr.Invoke$.invoke(invoke.scala:122)
    at sparklyr.StreamHandler$.handleMethodCall(stream.scala:97)
    at sparklyr.StreamHandler$.read(stream.scala:62)
    at sparklyr.BackendHandler.channelRead0(handler.scala:52)
    at sparklyr.BackendHandler.channelRead0(handler.scala:14)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
    at java.lang.Thread.run(Thread.java:745)
Joseph Steinke
  • 1
  • Could you try updating to the latest version of `sparklyr`? – kevinykuo Nov 09 '17 at 04:43
  • Ah, this is a bug due to incompatibility with earlier versions of Spark. I'll work on a fix. For the time being, the workaround is to use Spark 2.2.0 or use CRAN `sparklyr`. – kevinykuo Nov 09 '17 at 20:37
  • Tracking issue at https://github.com/rstudio/sparklyr/issues/1113 – kevinykuo Nov 09 '17 at 21:06
  • Fixed in devel: `devtools::install_github("rstudio/sparklyr")` – kevinykuo Nov 16 '17 at 15:54
  • Thank you @kevinykuo. Apologies for not responding. I created a second cluster, and it worked with Spark 2.2. This fix will be helpful on the original cluster, where I am stuck at 2.1 for the foreseeable future. – Joseph Steinke Nov 17 '17 at 16:34

0 Answers0