How to integrate the visualisation tool Graphexp with Neptune from EC2 instance in the same VPC as Neptune

Question

I have setup an AWS Neptune database cluster with one primary and two replica nodes in three private subnets, each in three availability zones within the same region. I have also created corresponding public subnets where EC2 instances will have a graph db visualizer app like https://github.com/bricaud/graphexp or https://github.com/erandal/graphexp. I am using the later (erandal), due to its more appealing UI. It is also deployed in apache httpd web server.

The private subnets have access to the internet through a NAT gateway associated with the public subnets.

For now I just have one EC2 instance to try out the connectivity of these UI tools to Neptune first before increasing the availability of the instances by deployment replicas in different AZs.

I have tried to connect by both options that Graphexp exposes (websockets and http) but without success. I keep getting the error - ERR_ADDRESS_UNREACHABLE in the Chrome Dev tools console. I have tried connecting to both the Neptune clusters' cluster endpoint hostname and its internal IP address etc. What is interesting is that I am able to call Neptune's gremlin endpoint successfully from the EC2 instance's shell using cURL. Why is this web app not working?

Answer 1

I've used GraphExp seamlessly with Neptune. The main catch in the setup is that the database calls are made over AJAX, which means that you need to setup your network connectivity (security groups) in such a way that your client instance that runs the browser has access to talk to the Neptune endpoint.

In your case, you mention that you setup your SGs in such a way that your EC2 instance is able to make CURL requests to the DB successfully. After you hosted GraphExp, how were you opening the page in the browser? Did you make the graphexp endpoint accessible outside of your VPC and connect to it directly from some other network? If yes, then your client instance (the one which has the browser page open) needs to be able to talk to the DB endpoint. Try doing a CURL from that instance into the DB endpoint and see if it works.

If you confirm that the above hypothesis is true, one way to fix it is by creating an ALB which is backed by your Neptune DB endpoints (IPs). And then attach a security group to the ALB that makes it accessible from your client machine. Now, make GraphExp talk to the ALB endpoint instead of the actual Neptune endpoint. I ended up using this in my experiments. A minor hiccup for me was that I did not get websocket to work. HTTP was good enough for me and I did not bother trying to debug what was going on with WS.

Once you get everything to work, let me know if you need to know other hacks to make stuff work better. For example, I ended up disabling some code in GraphExp that used to issue heavy read queries that I was not really interested in. That made the app load faster for my demo.

Hope this helps.

Answer 2

I wanted to host GraphEXP on the EC2 instance inside the VPC so that our team could see the graph visualisations. In the end I setup an apache server and hosted a simple php script that did a curl to the Neptune endpoint, that way I didn't have to create a lambda endpoint accessible from the internet (EC2 domain is behind Cloudflare access). Below is the simple php api I wrote.

<?php
header("Content-Type:application/json");
$url = $_POST['server_url'];
$query = $_POST['query'];
$queryObject = new stdClass();
$queryObject->gremlin = $query;
$curl = curl_init();

curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_POSTFIELDS, json_encode($queryObject));
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_HTTPHEADER, array(
    'Content-Type: application/json',
));
curl_setopt($curl, CURLOPT_URL, $url . "/gremlin");
$result = curl_exec($curl);
echo $result;

I also had to modify the GraphEXP graphioGremlin.js scripts file inside the run_ajax_request function I replaced the $.ajax param url with the path to my php api and then I also commented out the contentType and changed the data value to {query: gremlin_query, server_url} So it looked like the below

type: "POST",
url: [path to my api php file],
timeout: REST_TIMEOUT,
data: {query: gremlin_query, server_url}

Inside the graphConf.js I have the host value set to my Neptune endpoint without the port. Then in graphEXP ui I am choosing the REST option and everything works. Hopefully will help someone else...