Apache Cassandra 2.1.11 cluster failover does not work with datastax php driver

Question

I'm new to cassandra but have a background of Nosql and high availability with other technologies. A few days ago I installed Cassandra 3.0 but it sounds Datastax php driver does not support it at all! So I downgraded to the 2.1.11 version and wrote a simple php script to query it using Datastax php-driver and it is working as it should be.

<?php

try {
    $cluster   = Cassandra::cluster()
                    ->withContactPoints('127.0.0.1')
                     ->build();
    $keyspace  = 'mykeysssspace';
    $session   = $cluster->connect($keyspace);
    $statement = new Cassandra\SimpleStatement('SELECT userid, created_date, email FROM users');
    $future    = $session->executeAsync($statement);
    $result    = $future->get();
    foreach ($result as $row) {

        printf("userId: %s, email: %s\n", $row['userid'], $row['email']);
    }
}catch(Exception $e) {

    print $e->getMessage();
}
?>

Then I started two instance of Cassandra in the same machine (Ubuntu 11.10!) using different ip addresses. Then changed php code to:

$cluster   = Cassandra::cluster()
                    ->withContactPoints('127.0.0.1', '127.0.0.2')
                     ->build();

The problem is, as long as both instances are running everything works as expected and php outputs expected results. But when I stop one of the instances (no matter which) it gives

All hosts in current policy attempted and were either unavailable or failed

Even if I change it back to

$cluster   = Cassandra::cluster()
                    ->withContactPoints('running_instance_ip')
                     ->build();

it gives me the same error. But cqlsh works in every scenario.

Is it something about the Datastax php driver?

score 1 · Answer 1 · answered Nov 29 '15 at 00:42

1

What replication factor are you using on your keyspace? If you're using RF=1, the error may be indicating that there aren't sufficient hosts alive to fulfill all ranges. Raising the replication factor would help solve that.

answered Nov 29 '15 at 00:42

Jeff Jirsa

4,391
11
24

Thank you Jeff for the answer, but have tested it with the c code using c/c++ driver a few hours ago and its working fine! The problem is obviously with the php-deriver itself. Yes I'm using RF=1 but it does not seem to be the point – madz Nov 29 '15 at 01:15
Have you tested failover with the datastax-php-driver yet? Is it working fine with you? It would be great for me to know cause I'm digging into the driver right now – madz Nov 29 '15 at 01:18
As far as I know, the datastax PHP driver uses the datastax C++ driver - it's just a very thin wrapper. If C++ works, I'd expect PHP to work as well. If that's not behaving, then I'm at a loss. – Jeff Jirsa Nov 29 '15 at 01:39
I totally agree with you and that is exactly where I'm confused – madz Nov 29 '15 at 20:42
Are the hosts in the same DC? The datastax drivers collectively have pluggable/configurable load host-choosing policies ( http://datastax.github.io/cpp-driver/topics/configuration/ ) , I wonder if the PHP driver is defaulting to something unexpected – Jeff Jirsa Nov 30 '15 at 07:41
Yep they are in the same DC But I already tested those configurations with no success. Thinking of your answer again and wondering why did you mention of RF=1 could cause the problem? Considering I only have 2 nodes in the cluster, why would you think making RF to a higher value could solve my problem? (yep I already tested RF=1 topology with the c driver and it worked for failover, but I want to know the rational behind your answer to see if I'm missing something) – madz Dec 01 '15 at 00:10
With RF=1, one node owns each partition - if that's the node you stop, there is no failover possible. Upping replication factor will allow Cassandra to replicate that partition to multiple hosts so a host remains available to respond to the query when another host goes offline – Jeff Jirsa Dec 02 '15 at 07:49
Thank you for the comment. Theoretically you're right. But I have no idea why with cqlsh and RF=1 even when I stop the node that owns owns the partition, other works as well! Anyway problem solved but I have no idea why. I changed Keyspace's class to NetworkTopologyStrategy and it worked well! I also changed RF=2 as you mentioned (did not test if with RF=1 works or not). Thank you so much for helping. Please edit your answer about having NetworkTopologyStrategy to make it complete so I can accept. Thanks – madz Dec 02 '15 at 13:50
The topology strategy change is irrelevant. The fix is RF2. NTS may work due to luck (or you changed how you test), but that is unlikely to be the real fix. – Jeff Jirsa Dec 02 '15 at 22:17
This is exactly what I've done to test it. Create two instance of cassandra 2.1.11 (both seeds) - call them n1 and n2 - and created a KS with RF=1 and ST using cqlsh (cqlsh was connected to n1). Then created a table and inserted some rows of data. Then stopped n1 and connected cqlsh to n2 and tested if failover works or not and it worked. Then tested again with php driver and it did not! Then tested the same with the c driver and it worked. Then changed to NTS and changed RF=2 (your right here about changing my test and I mentioned it in my comment) and it worked. – madz Dec 03 '15 at 02:45
The question is why with the RF=1 cqlsh and c driver's failover worked in the first place and with the php driver it did not. I suspected to a consistency problem here. But changing the consistency to the lowest one did not work with the php either. But with the NTS worked. If your right and RF should be 2 for testing two node cluster, it should not be worked in the first place. It is also very good that you don't give up your knowledge base over some random guy's test ;) – madz Dec 03 '15 at 02:51
What I'm not explaining very well is that with RF=1, only 1 of the 2 nodes will have any given key. What I'm hinting at, but not explicitly stating, is that I suspect you tried a different key, which hashed to the OTHER node, so that when you powered down one of the 2 nodes in the cluster, you powered down the OWNING node in one test and the NON-OWNING node in the other That is, I suspect your test isn't identical between c++ and php, and that the hash function is placing one key on one node, and the different key in the other node, so when you turn off a node, it's having different behavior – Jeff Jirsa Dec 03 '15 at 21:03
Ok well, maybe I don't undrestand cassandra's hashing very well, but had inserted a single row via cqlsh a day before setting up any driver for c or php. Having a single row inserted via cqlsh does not seem to have such a key behavior to me. Anyway as I mentioned before, theoretically you are right about RF, but for some reason its working strange for me. I give a + for your answer so others have this very issue take your advice and try with higher RF – madz Dec 05 '15 at 21:23

score 0 · Answer 2 · answered Apr 17 '16 at 02:27

In my case, I was running a test in an isolated environment with ONE node, but I was setting up the context with a replication factor of 2. So certain functions would fail with a timeout since it could never finish the wait for the other non-existent nodes to be updated in time.

So one solution is to make sure your replication factor is no larger than the number of nodes in your cluster (and yes, I fail to understand why Cassandra cannot just tell us: RF too large as an error.)

Thank you Alexis. Your totally right for your situation. But my problem was the wired behavior among different drives and the clqsh itself . — madz, Apr 19 '16 at 17:55

score 0 · Answer 3 · edited Jul 25 '17 at 14:12

You can try Retry policy

$retry_policy = new Cassandra\RetryPolicy\DowngradingConsistency();
$cluster     = Cassandra::cluster()
                 ->withContactPoints('127.0.0.1','127.0.0.2')
                 ->withRetryPolicy(new Cassandra\RetryPolicy\Logging($retry_policy))
                 ->build();
$session     = $cluster->connect("simplex");
$statement   = $session->prepare("INSERT INTO playlists (id, song_id, artist, title, album)
                                 VALUES (62c36092-82a1-3a00-93d1-46196ee77204, ?, ?, ?, ?)");
$options     = array(
                    'consistency' => Cassandra::CONSISTENCY_ONE,
                    );
$session->execute($statement, $options);

Working for me.

Apache Cassandra 2.1.11 cluster failover does not work with datastax php driver

3 Answers3