1

I follow the guide to deploy the spark on Alluxio. When I try to load data from alluxio to run rdd operation,

val ccc = sc.textFile("alluxio://localhost:19998/findbugs.xml")
ccc.count

error shows up like following:

16/07/24 23:27:16 INFO logger.type: getFileStatus(alluxio://localhost:19998/findbugs.xml)
16/07/24 23:27:16 ERROR logger.type: Frame size (273247862) larger than max length (16777216)!
alluxio.org.apache.thrift.transport.TTransportException: Frame size (273247862) larger than max length (16777216)!
    at alluxio.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
    at alluxio.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    at alluxio.org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)

java version is 1.8.0_74, spark:1.2.0; alluxio 1.2.0
The issue bothers me for such a long time.

dtolnay
  • 9,621
  • 5
  • 41
  • 62
Carl H
  • 405
  • 1
  • 8
  • 20

1 Answers1

1

The typical reason for this error is when the incorrect port is used to connect to the Alluxio master. The default port is 19998. You could try restarting spark-shell and connecting to the correct port.

Another possible reason for this error is if the client and master have different security settings. The authentication type (configuration parameter alluxio.security.authentication.type) should match on the client and the master. Here is information on the configuration parameters: http://www.alluxio.org/docs/master/en/Configuration-Settings.html#security-configuration

Gene Pang
  • 231
  • 1
  • 5