0

I configured two DC with replication in two regions (NCSA and EMEA) using Janusgraph (Gremlin/Cassandra/Elasticsearch). The replication work well and everything, however the performance are not that great.

I get time of around 250ms just for a read on a node on NCSA (vs 30ms when I have only one 1 DC / 1 Node) and for a write it is around 800ms.

I tried to modify some configuration:

  • storage.cassandra.replication-factor
  • storage.cassandra.read-consistency-level
  • storage.cassandra.write-consistency-level

Is there any other settings/configurations that I could modify in order to get better performance for a multi-region setup or that kind of performance is expected with Janusgraph/Cassandra?

Thanks

The lowest time I was able to get were with

  • storage.replication-strategy-class=org.apache.cassandra.locator.NetworkTopologyStrategy
  • storage.cassandra.replication-factor=6
  • storage.cassandra.read-consistency-level=ONE
  • storage.cassandra.write-consistency-level=ONE
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns (effective)  Host ID                               Rack
UN  10.130.xxx.xxx  184.02 KB  256     100.0%            7c4c23f4-0112-4023-8af1-81a179f68973  RAC2
UN  10.130.xxx.xxx  540.67 KB  256     100.0%            193f0814-649f-4450-8b2e-85344f2c3cf2  RAC3
UN  10.130.xxx.xxx  187.47 KB  256     100.0%            fbbc42d6-a061-4604-935e-dbe1155d4017  RAC1
Datacenter: DC2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens  Owns (effective)  Host ID                               Rack
UN  10.30.xxx.xxx    93.3 KB    256     100.0%            e7221808-ccb4-414a-b5b6-6e578ecb6f25  RAC3
UN  10.30.xxx.xxx    287.62 KB  256     100.0%            ca868262-4b5d-44d6-80f9-25439f8d2611  RAC2
UN  10.30.xxx.xxx    282.27 KB  256     100.0%            82d0f75d-635c-4016-84ca-ef9d1afda066  RAC1
DBrouillard
  • 1
  • 1

1 Answers1

0

Janusgraph comes with different caches levels, activate some of them may help.

About ConsistencyLevel, in a multi-dc configuration LOCAL_xxx values will provide better performances but for safety I will initialize the name of the local or closest Cassandra datacenter. (configuration parameter : storage.cassandra.astyanax.local-datacenter)

Are you able to say where the time is spent (in the Cassandra layer on in the JanusGraph layer)? To know what is the response time of Cassandra, you can run nodetool proxyhistograms which shows the full request latency recorded by the coordinator.

Leleu Eric
  • 31
  • 3
  • I forgot to mention that I'm currently `storage.backend=cassandrathrift`. If I change it by `storage.backend=astyanax`, gremlin-server is not able to connect anymore, so I guess I will need to change some configurations in Janusgraph in order to use "astyanax". Do you think "astyanax" will have better performance in a multi-region setup vs cassandrathrift? Concerning the time, I ran `nodetool proxyhistograms` on one machine in NCSA and EMEA. – DBrouillard May 31 '18 at 14:40
  • NCSA `proxy histograms` `Percentile Read Latency Write Latency Range Latency` `(micros) (micros) (micros)` `50% 88148.00 88148.00 105778.00` `75% 88148.00 88148.00 105778.00` `95% 152321.00 182785.00 152321.00` `99% 152321.00 182785.00 152321.00` `Min 150.00 1332.00 73458.00` `Max 152321.00 182785.00 152321.00` – DBrouillard May 31 '18 at 14:54
  • EMEA `proxy histograms` `Percentile Read Latency Write Latency Range Latency` `(micros) (micros) (micros)` `50% 88148.00 88148.00 105778.00` `75% 88148.00 88148.00 182785.00` `95% 182785.00 88148.00 182785.00` `99% 182785.00 88148.00 182785.00` `Min 73458.00 2300.00 73458.00` `Max 182785.00 88148.00 182785.00` – DBrouillard May 31 '18 at 14:55
  • I don't know if astyanax is more efficient that cassandrathrift. – Leleu Eric Jun 01 '18 at 12:56
  • It depends of the amount of data, the hardware and the load on your cluster but if you have a lot of data with a consequent workload coordinator latencies are not so hight.To see latency on R/W operation on each node, use the `nodetool cfhistograms` on Janusgraph tables. – Leleu Eric Jun 01 '18 at 13:03