1

I have a strange error on Alluxio with Spark. I read 20.000 files with Spark from Alluxio and it works. But I read 40.000 files with Spark from Alluxio and it does'n work. I use Alluxio 1.2, Spark 1.6.0 and I read data with file API: FileSystem fs = FileSystem.Factory.get(); AlluxioURI path = new AlluxioURI(/partition0); ...

16/08/19 16:08:40 INFO logger.type: Client registered with FileSystemMasterClient master @ master/127.0.0.1:19998
16/08/19 16:08:41 ERROR logger.type: Frame size (17277505) larger than max length (16777216)!
org.apache.thrift.transport.TTransportException: Frame size (17277505) larger than max length (16777216)!
        at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
        at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.protocol.TProtocolDecorator.readMessageBegin(TProtocolDecorator.java:135)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
        at alluxio.thrift.FileSystemMasterClientService$Client.recv_listStatus(FileSystemMasterClientService.java:503)
        at alluxio.thrift.FileSystemMasterClientService$Client.listStatus(FileSystemMasterClientService.java:489)
        at alluxio.client.file.FileSystemMasterClient$8.call(FileSystemMasterClient.java:220)
        at alluxio.client.file.FileSystemMasterClient$8.call(FileSystemMasterClient.java:216)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:324)
        at alluxio.client.file.FileSystemMasterClient.listStatus(FileSystemMasterClient.java:216)
        at alluxio.client.file.BaseFileSystem.listStatus(BaseFileSystem.java:195)
        at alluxio.client.file.BaseFileSystem.listStatus(BaseFileSystem.java:186)
        at Main.main(Main.java:119)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" java.io.IOException: Failed after 32 retries.
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:334)
        at alluxio.client.file.FileSystemMasterClient.listStatus(FileSystemMasterClient.java:216)
        at alluxio.client.file.BaseFileSystem.listStatus(BaseFileSystem.java:195)
        at alluxio.client.file.BaseFileSystem.listStatus(BaseFileSystem.java:186)
        at Main.main(Main.java:119)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

It's not a alluxio.security.authentication.type problem because I run Alluxio on local and Alluxio master address is correct. I don't understand why it's not work with 40.000 files whereas it's work with 20.000 files.

I have also modified alluxio.network.thrift.frame.size.bytes.max but not result..

dtolnay
  • 9,621
  • 5
  • 41
  • 62
TiGi
  • 37
  • 8

1 Answers1

0

This problem can be caused by different possible reasons :

  • Please double-check if the port of Alluxio master address is correct. The default listening port for Alluxio master is port 19998, while a common mistake causing this error message is due to using a wrong port in master address(e.g., using port 19999 which is the default Web UI port for Alluxio master).

  • Please ensure that the security settings of Alluxio client and master are consistent. Alluxio provides different approaches to authenticate users by configuring alluxio.security.authentication.type. This error happens if this property is configured with different values across servers and clients (e.g., one uses the default value NOSASL while the other is customized to SIMPLE). Please read Configuration-Settings for how to customize Alluxio clusters and applications.

  • config between Apache-Spark and Alluxio. You must change the JVM environnement of Spark to work alluxio.network.thrift.frame.size.bytes.max in alluxio/conf/alluxio-site.properties. To do that, you must add export SPARK_CLASSPATH=${ALLUXIO_HOME}/conf:${SPARK_CLASSPATH} in spark-env.sh or add --driver-class-path pathAlluxio/conf with spark-submit command

To me, it was the third solution

TiGi
  • 37
  • 8