Cassandra another frame size issue

Question

I know that this kind of issue with Apache Cassandra was a while ago for example: Pentaho Frame size (17727647) larger than max length (16384000)! and thrift_max_message_length_in_mb not recognized Cassandra

but my problem is a little bit different. I use Cassandra v2.0.7 and I want insert data from Pentaho Kettle (5.1) to Cassandra. I have to enable encryption (SSL), so I chaged cassandra.yaml

client_encryption_options:
enabled: true
keystore: /usr/cassandra/cassandra/conf/.keystore
keystore_password: password
require_client_auth: true
# Set trustore and truststore_password if require_client_auth is true
truststore: /usr/cassandra/cassandra/conf/.truststore
truststore_password: password

Now, there is a strange situation because, when I disable encryption (enabled: false) and insert data by Pentaho everything works well.But when I enable encryption I get this:

   2015/01/05 17:42:38 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : A problem occurred during initializ
ation of the step
2015/01/05 17:42:38 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : org.apache.thrift.transport.TTransp
ortException: Frame size (352518400) larger than max length (16384000)!
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.cassandra.thrift.Cassandra$Client.recv_set_cql_version(Cassandra.java:1855)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.apache.cassandra.thrift.Cassandra$Client.set_cql_version(Cassandra.java:1842)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.checkOpen(CassandraConnection.java:159)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.setKeyspace(CassandraConnection.java:174)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.LegacyKeyspace.setKeyspace(LegacyKeyspace.java:100)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.getKeyspace(CassandraConnection.java:277)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.initialize(CassandraOutput.java:218)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:353)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2015/01/05 17:42:38 - Cassandra Output.0 -      at java.lang.Thread.run(Thread.java:745)
java.lang.NullPointerException
        at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:356)
        at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
        at java.lang.Thread.run(Thread.java:745)
2015/01/05 17:42:38 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Unexpected error
2015/01/05 17:42:38 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : java.lang.NullPointerException
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:356)
2015/01/05 17:42:38 - Cassandra Output.0 -      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2015/01/05 17:42:38 - Cassandra Output.0 -      at java.lang.Thread.run(Thread.java:745)
child index = 1, logging object : org.pentaho.di.core.logging.LoggingObject@47406edf parent=277afd35-cec6-4972-a572-a68a58ff9ae7
2015/01/05 17:42:38 - t_product_rb - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!
2015/01/05 17:42:38 - t_product_rb - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!

I insert data with CQL3. Now, I found that in cassandra.yaml there are two property : -thrift_framed_transport_size_in_mb and -thrift_max_message_length_in_mb <- but in my file I don't have this

I check a few configuration with this property. Once I added this thrift_max_message_length_in_mb , then I remove this etc. But everytime I got this error.

I notice that encryption add something BIG to my frame, but I don't know what and how. Has somebody know how to fix this?

Some other info about this problem: https://issues.apache.org/jira/browse/THRIFT-1324 https://issues.apache.org/jira/browse/THRIFT-1323

EDIT I noticed again that when I change port from 9160 to 9042 (use CQL3) I get different error.

2015/01/09 12:53:21 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : A problem occurred during initializ
ation of the step
2015/01/09 12:53:21 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : org.apache.thrift.transport.TTransp
ortException: java.net.SocketTimeoutException: Read timed out
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.cassandra.thrift.Cassandra$Client.recv_set_cql_version(Cassandra.java:1855)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.cassandra.thrift.Cassandra$Client.set_cql_version(Cassandra.java:1842)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.checkOpen(CassandraConnection.java:159)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.setKeyspace(CassandraConnection.java:174)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.LegacyKeyspace.setKeyspace(LegacyKeyspace.java:100)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.cassandra.legacy.CassandraConnection.getKeyspace(CassandraConnection.java:277)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.initialize(CassandraOutput.java:218)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:353)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.lang.Thread.run(Thread.java:745)
2015/01/09 12:53:21 - Cassandra Output.0 - Caused by: java.net.SocketTimeoutException: Read timed out
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.net.SocketInputStream.socketRead0(Native Method)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.net.SocketInputStream.read(SocketInputStream.java:152)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.net.SocketInputStream.read(SocketInputStream.java:122)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
2015/01/09 12:53:21 - Cassandra Output.0 -      ... 18 more
java.lang.NullPointerException
        at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:356)
        at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
        at java.lang.Thread.run(Thread.java:745)
2015/01/09 12:53:21 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Unexpected error
2015/01/09 12:53:21 - Cassandra Output.0 - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : java.lang.NullPointerException
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.steps.cassandraoutput.CassandraOutput.processRow(CassandraOutput.java:356)
2015/01/09 12:53:21 - Cassandra Output.0 -      at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2015/01/09 12:53:21 - Cassandra Output.0 -      at java.lang.Thread.run(Thread.java:745)
child index = 4, logging object : org.pentaho.di.core.logging.LoggingObject@6a4ea42b parent=c8d41836-602c-4a17-8977-62d691c419c5
2015/01/09 12:53:21 - Cassandra Output.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)
2015/01/09 12:53:21 - t_customer_cm - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!
2015/01/09 12:53:21 - t_customer_cm - Transformation detected one or more steps with errors.
2015/01/09 12:53:21 - t_customer_cm - Transformation is killing the other steps!
2015/01/09 12:53:21 - Dummy (do nothing).0 - Finished processing (I=0, O=0, R=10002, W=10002, U=0, E=0)
2015/01/09 12:53:21 - Get System Info.0 - Finished processing (I=0, O=0, R=20003, W=20003, U=0, E=0)
2015/01/09 12:53:21 - t_customer_cm - ERROR (version 5.1.0.0, build 1 from 2014-06-19_19-02-57 by buildguy) : Errors detected!

On cassandra I get this:

ERROR 12:53:11,807 Unexpected exception during request
 org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 00000028800100010000000f7365745f63716c5f76657273696f6e000000010b0001000000
5332e302e3100
        at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:871)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

Any ideas?

JensG · Answer 1 · 2015-01-05T17:15:30.063

Frame size (352518400) larger than max length (16384000)!
I notice that encryption add something BIG to my frame, but I don't know what and how. Has somebody know how to fix this?

With regard to the frame size it does not really matter what it is, only how big it is. As Bryan wrote in THRIFT-1324, the only difference in size between framed and "unframed" is a constant 4 bytes (int 32), preceding the data block and holding its size.

The theoretical solution seems very obvious to me: Find out, how big your biggest frame is and configure the settings accordingly. Each request and each response must fit into that.

Theoretical, because it seems strange that enabling encryption adds 21x the size to your data (you said it works without, so it must be <= 16384000). That sounds very unusual.

Andy Tolbert · Answer 2 · 2015-01-05T17:15:42.350

0

Check to make sure whether you are connecting with the CQL binary protocol (port 9042) using unsecured. The stack trace looks to indicate that Thrift is being used, but CQL3 uses a completely different interface. Is it possible that when using SSL it's using the thrift protocol (port 9160) and when using unsecured it's using the binary protocol (port 9042)?

EDIT: This answer may not be a good one. Your mention of using CQL3 got me thinking that thrift shouldn't even be involved, still might be worth taking a look to see if the unsecured version is using the native binary protocol or thrift.

edited Jan 05 '15 at 17:15

answered Jan 05 '15 at 17:07

Andy Tolbert

11,418
1
30
45

1

I notice that I use 9160 port, but in Pentaho I have "Use CQL version3". This configuration works well, when I insert data without SSL. With SSL, the same Pentaho configuration and port 9160 I get that error. – Kacu Jan 07 '15 at 11:58
Thanks for confirming, this is certainly not the problem then :) – Andy Tolbert Jan 07 '15 at 16:43

Cassandra another frame size issue

2 Answers2