janusgraph encounters `PermanentBackendException` while interacting with AWS opensearch version 7 and above

Question

We are trying to run Janusgraph version 0.6.2 using AWS opensearch(elasticsearch) version 7.10 as the indexing backend. Things work fine with version 6.x but when we try to connect to version 7.x we encounter the following exception.

org.janusgraph.diskstorage.PermanentBackendException: method [PUT], host [https://vpc-xxxxxx.us-east-2.es.amazonaws.com:443], URI [/_cluster/settings], status line [HTTP/1.1 401 Unauthorized] {"Message":"Your request: '/_cluster/settings' payload is not allowed."}

Janusgraph version info:

86   [main] INFO  org.janusgraph.graphdb.server.JanusGraphServer  - JanusGraph Version: 0.6.2
86   [main] INFO  org.janusgraph.graphdb.server.JanusGraphServer  - TinkerPop Version: 3.5.3

More detailed stack trace is below:

3115 [main] INFO  org.janusgraph.diskstorage.Backend  - Configuring index [search]
3387 [main] INFO  com.newforma.janusgraph.es.awsauth.AWSV4AuthHttpClientConfigCallback  - Initialized AWSV4AuthHttpClientConfigCallback for region us-east-2
3782 [main] WARN  org.apache.tinkerpop.gremlin.server.util.DefaultGraphManager  - Graph [graph] configured at [/etc/opt/janusgraph/janusgraph.properties] could not be instantiated and will not be available in Gremlin Server.  GraphFactory message: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class org.janusgraph.core.JanusGraphFactory]
    at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:84)
    at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:80)
    ... 14 more
Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.es.ElasticSearchIndex
    at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:79)
    at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:527)
    at org.janusgraph.diskstorage.Backend.getIndexes(Backend.java:511)
    at org.janusgraph.diskstorage.Backend.<init>(Backend.java:239)
    at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:127)
    ... 19 more
Caused by: org.janusgraph.diskstorage.PermanentBackendException: method [PUT], host [https://vpc-xxxxxx.us-east-2.es.amazonaws.com:443], URI [/_cluster/settings], status line [HTTP/1.1 401 Unauthorized]
{"Message":"Your request: '/_cluster/settings' payload is not allowed."}
    at org.janusgraph.diskstorage.es.ElasticSearchIndex.setupMaxOpenScrollContextsIfNeeded(ElasticSearchIndex.java:445)
    at org.janusgraph.diskstorage.es.ElasticSearchIndex.<init>(ElasticSearchIndex.java:388)
    ... 32 more
Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [https://vpc-xxxxxx.us-east-2.es.amazonaws.com:443], URI [/_cluster/settings], status line [HTTP/1.1 401 Unauthorized]
{"Message":"Your request: '/_cluster/settings' payload is not allowed."}
    at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
    at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:482)
    at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.performRequest(RestElasticSearchClient.java:473)
    at org.janusgraph.diskstorage.es.rest.RestElasticSearchClient.updateClusterSettings(RestElasticSearchClient.java:269)
    at org.janusgraph.diskstorage.es.ElasticSearchIndex.setupMaxOpenScrollContextsIfNeeded(ElasticSearchIndex.java:443)

score 3 · Accepted Answer · answered Sep 05 '22 at 05:14

From the stack trace it appears that janusgraph was trying to set a high value for the elasticsearch property max_open_scroll_context. It is 500 by default.

AWS opensearch(elasticsearch) 7.x onwards doesn't let us set cluster properties. Tried the following from kibana and I was able to get a similar response. This operation was supported in AWS managed elasticsearch 6.x version.

PUT _cluster/settings
{
    "persistent" : {
        "search.max_open_scroll_context": 1024
    },
    "transient": {
        "search.max_open_scroll_context": 1024
    }
}

401 - Unauthorized
{"Message":"Your request: '/_cluster/settings' payload is not allowed."}

We can disable setting max_open_scroll_context property while janugraph starts by setting the property index.[x].elasticsearch.setup-max-open-scroll-contexts to false.

You can read more on this in configuration reference section on elasticsearch https://docs.janusgraph.org/configs/configuration-reference/#indexxelasticsearch

janusgraph encounters `PermanentBackendException` while interacting with AWS opensearch version 7 and above

1 Answers1