3

Context

I'm using Titan v1.0.0, on AWS infrastructure and want to support failover/fault tolerance. AWS will take care of DynamoDb storage backend but it seems necessary to have several titan instances serviced by an (ELB) load balancer.

I'm using a nodeJs library to get to gremlin, and gremlin to access Titan.

Question

So, how does the Titan (not backend storage) clustering work? If at all.

To be clear, I'm not talking about any backend storage clustering, as I'm using dynamoDb on AWS. The documentation on transaction locking suggests to me that a titan cluster must exist, as other titan nodes wouldn't know about the locking without some sort of inter-communication. But I don't see any configuration options that support this.

If clustering is possible on titan, does anyone have any information on how to get this setup in a production High-Availability setup?

An illustration of the server-side architecture:

[NodeJsA]\             /[TitanA]\
          \           /          \
           [ELB (AWS)]            [DynamoDb (AWS)]
          /           \          /
[NodeJsB]/             \[TitanB]/

Further, if there is no clustering of the titan nodes then a change made via the TitanA node (above) could take the following amount of time to be seem on TitanB node (worst case):

(AWS eventual consistency convergence time (~1 sec) + TitanB cache timeout + poll time from NodeJs nodes to titanDB)

Another consequence of the lack of clustering would be that sessions would have be to pinned in the ELB else a read request, after an update, could be served by a different node with stale information.

John Stephenson
  • 499
  • 4
  • 13

1 Answers1

2

Titan doesn't do any "clustering" outside of what is supported by the selected backend. You referred to "locking" as something that would indicate that clustering is supported, but if you read about the locking providers in that link you supplied you'll see that locking isn't really doing anything terribly fancy at a Titan level and that it is backend dependent. So Titan instances really don't have any external clustering capabilities or knowledge of each other. You therefore need to take that into account with respect to your architecture.

stephen mallette
  • 45,298
  • 5
  • 67
  • 135
  • Many thanks for the clarification. I must have read that page about 10 times, and the part where it says locking is abstracted must have gone past me each time. Follow up question... do you have any information about AWS locking providers? Does AWS provide the type of locking that is described in the link, and if so how do you consume it/know you are consuming it correctly? – John Stephenson Jan 16 '17 at 19:05
  • I am not well informed on the aws implementation, so I'm not sure if they have their own locking provider. All i can say is that I've never heard mention of one in the Titan community. – stephen mallette Jan 16 '17 at 20:05
  • Many thanks :-). For completeness (I'm hoping not to need to do this) but if I had to implement a locking provider for AWS is there a guide or an API reference that I could look at, or something along those lines? – John Stephenson Jan 17 '17 at 08:28
  • I don't think so. It wasn't a "feature" commonly reached for imo so the demand to document it was fairly low. I suppose you'd want to start by looking [here](https://github.com/thinkaurelius/titan/tree/ee226e52415b8bf43b700afac75fa5b9767993a5/titan-core/src/main/java/com/thinkaurelius/titan/diskstorage/locking) – stephen mallette Jan 23 '17 at 11:44
  • @JohnStephenson We use the same architecture as you but are having issues with an sshtunnel from the nodejs instances trying to connect to the Titan instances via the ELB. We ended up coding a custom Ping but that seems unduly burdensome to keep the websocket connection alive from the nodejs intances to the Titan instances through the ELB. – Vijay Mar 27 '17 at 03:04
  • @Vijay - Yikes! I've stepped away from this in the last month or two, but starting to return to it now. Good to know. If you have had any simplifications/developments/revelations please share :-). – John Stephenson May 17 '17 at 23:06