jclouds connection refused during hazelcast clustering

Question

Questions

Why does it use localhost?
What does keystone have to do with it?
- I can't seem to configure a keystone endpoint

Context

App: Spring Boot (1.5.6) REST API
Hibernate 5.2
Hazelcast 3.9 - as 2nd-level cache only
hazelcast-jclouds 3.7.1
jclouds-compute and jclouds-allcompute 2.0.2
Openstack cloud for VMs running the app

The Setup

I have my hazelcast.xml configured as follows:

        <discovery-strategies>
            <discovery-strategy class="com.hazelcast.jclouds.JCloudsDiscoveryStrategy" enabled="true">
                <properties>
                    <property name="modules">org.jclouds.logging.slf4j.config.SLF4JLoggingModule</property>
                    <property name="provider">openstack-nova</property>
                    <property name="endpoint">http://dev.nova.cloud.youdontknow.net:8774/v2/</property>
                    <property name="identity">redacted</property>
                    <property name="credential">cens0red</property>
                </properties>
            </discovery-strategy>
        </discovery-strategies>

The problem

App initialization fails. Here's some log tidbits:

[TRACE] o.j.r.internal.RestAnnotationProcessor   : looking up default endpoint for org.jclouds.openstack.keystone.v2_0.AuthenticationApi.public abstract org.jclouds.openstack.keystone.v2_0.domain.Access org.jclouds.openstack.keystone.v2_0.AuthenticationApi.authenticateWithTenantNameAndCredentials(java.lang.String,org.jclouds.openstack.keystone.v2_0.domain.PasswordCredentials)[bnet-web, PasswordCredentials{username=redacted, password=*****}]
[TRACE] o.j.r.internal.RestAnnotationProcessor   : using default endpoint Optional.of(http://localhost:5000/v2.0/) for org.jclouds.openstack.keystone.v2_0.AuthenticationApi.public abstract org.jclouds.openstack.keystone.v2_0.domain.Access org.jclouds.openstack.keystone.v2_0.AuthenticationApi.authenticateWithTenantNameAndCredentials(java.lang.String,org.jclouds.openstack.keystone.v2_0.domain.PasswordCredentials)[bnet-web, PasswordCredentials{username=redacted, password=*****}]
[TRACE] o.j.rest.internal.InvokeHttpMethod       : << converted AuthenticationApi.authenticateWithTenantNameAndCredentials to POST http://localhost:5000/v2.0/tokens HTTP/1.1

And here's bits of the exception stack traces:

Caused by: com.hazelcast.core.HazelcastException: Failed to get registered addresses
at com.hazelcast.jclouds.JCloudsDiscoveryStrategy.discoverNodes(JCloudsDiscoveryStrategy.java:93)
at com.hazelcast.jclouds.JCloudsDiscoveryStrategy.discoverLocalMetadata(JCloudsDiscoveryStrategy.java:106)
at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverLocalMetadata(DefaultDiscoveryService.java:91)

...

Caused by: org.jclouds.http.HttpResponseException: Connection refused: connect connecting to POST http://localhost:5000/v2.0/tokens HTTP/1.1
    at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:122)
    ...
    at com.sun.proxy.$Proxy147.authenticateWithTenantNameAndCredentials(Unknown Source)
    at org.jclouds.openstack.keystone.v2_0.functions.AuthenticatePasswordCredentials.authenticateWithTenantName(AuthenticatePasswordCredentials.java:43)

Other Notes

Looks like it's using the default keystone address configured in org.jclouds.openstack.keystone.v2_0.KeystoneApiMetadata - but I don't know how that's involved.

score 2 · Accepted Answer · answered Nov 20 '17 at 17:32

2

By looking at the code, I think hazlecast-jclouds is not prepared to manage generic APIs. When connecting to a provider, you don't need to specify the endpoint, as it is well-known (the AWS endpoints, Google, Azure, etc), but when using generic APIs such as OpenStack or CloudStack, you need to tell jclouds where to connect. Unfortunately, it looks like hazlecast-jclouds lacks support for configuring custom endpoints for generic APIs.

A quick look at the code suggests that it could be easy to add, though. The properties that are taken into account are defined in the JCloudsDiscoveryStrategyFactory, and then read in the ComputeServiceBuilder to create the jclouds context.

I'm not familiar with Hazlecast, but I'd say that adding the definition for the "endpoint" property, and then, if present, configuring it by calling the jclouds contextBuilder.endpoint(endpoitn) method should do the trick.

answered Nov 20 '17 at 17:32

Ignasi Barrera

1,476
9
8

However: I was following this readme: https://github.com/hazelcast/hazelcast-jclouds/blob/master/README.md Which has a link to providers (here: https://jclouds.apache.org/reference/providers/#compute) which includes `openstack-nova` - so it seems like it should be supported? Oh, btw, I think I maybe should be calling the property `openstack-nova.endpoint` **EDIT:** Nope, that didn't help. – Eric J Turley Nov 20 '17 at 17:53
I've just created a branch of `hazlecast-jclouds` with the fix I mentioned. Could you try this branch in my fork and see if it works? https://github.com/nacx/hazelcast-jclouds/tree/endpoint – Ignasi Barrera Nov 20 '17 at 18:13
I will give it a shot - but note that while I'm using that example for configuration, I'm not writing any code to directly access jclouds. I'm only configuring it in hazelcast for clustering, and using that as a hibernate 2nd-level cache. – Eric J Turley Nov 20 '17 at 20:34
Yes. My change will (hopefully) make hazlecast read the configured endpoint property instead of ignoring it. – Ignasi Barrera Nov 20 '17 at 20:38
Excellent. I built and installed the library and bumped my app's pom to `3.7.2-SNAPSHOT`. It now uses the specified endpoint. But there's an odd log which states: `TRACE 3896 --- [ restartedMain] o.j.r.internal.RestAnnotationProcessor : using default endpoint Optional.of(http://dev.nova.cloud.blizzard.net:8774/v2/) for org.jclouds.openstack.keystone.v2_0.AuthenticationApi` so that seems weird, since this is not the `default`. Now, I'm getting an auth failed error, so I believe that's progress :) – Eric J Turley Nov 20 '17 at 21:03
Good! I'll open a pull request then. Regarding the auth issue, for the "identity" use: `tenantname:username` – Ignasi Barrera Nov 20 '17 at 21:05
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/159427/discussion-between-eric-turley-and-ignasi-barrera). – Eric J Turley Nov 20 '17 at 21:10