Spring boot micro meter datadog socket connection error

Question

I am working on to create some custom metrics for my spring boot 2 rest api. I have added the required micro meter and datadog dependency. My office machine works behind a proxy. I have setup proxy through spring boot plugin.

-Dhttp.proxyHost=xxxx.proxy.com
-Dhttp.proxyPort=xxxx

below are in my application.properties file. management.metrics.export.datadog.apiKey=mykey

management.metrics.export.datadog.uri=https://app.datadoghq.com

management.metrics.export.datadog.enabled=true

management.metrics.export.datadog.step=10s

But I am getting the socket connection timeout.

   [datadog-metrics-publisher] 10 Apr 2020 16:51:39,552 WARN  DatadogMeterRegistry [{}]: java.net.SocketTimeoutException: connect timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:606)
    at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
    at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
    at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264)
    at io.micrometer.core.ipc.http.HttpUrlConnectionSender.send(HttpUrlConnectionSender.java:96)
    at io.micrometer.core.ipc.http.HttpSender$Request$Builder.send(HttpSender.java:284)
    at io.micrometer.datadog.DatadogMeterRegistry.publish(DatadogMeterRegistry.java:141)
    at io.micrometer.core.instrument.push.PushMeterRegistry.publishSafely(PushMeterRegistry.java:48)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

As far as I debugged the io.micrometer.core.ipc.http.HttpUrlConnectionSender.send method is failing and I dont under how the micro meter data dog takes the proxy details.

The micrometer doc says

management.metrics.export.datadog.uri=https://app.datadoghq.com # URI to ship metrics to. If you need to publish metrics to an internal proxy en-route to Datadog, you can define the location of the proxy with this.

But I dont understand what it means? should I replace this url with my proxy url or is there any specific uri pattern with the proxy? I am using spring boot 2.2.4.RELEASE

Facing the same issue and using the same spring boot version. I have tried increasing the timeout to `15s` and configuring `-Dhttps.proxyHost` as well, neither of the two helped. For me the service is running in AWS EC2. — dShringi, Apr 14 '20 at 15:10
I updated the logging level to trace and the issues seems to be intermittent. — dShringi, Apr 20 '20 at 07:59

score 1 · Accepted Answer · answered Apr 16 '20 at 10:00

If you are trying to connect to an HTTPS URL for Datadog (https://app.datadoghq.com in your example), then you will need to set the https.proxyHost system property for it to have effect - http.proxyHost is for HTTP URLs[1]. These are system-wide settings that will be used by the default HttpSender (HttpUrlConnectionSender) if a Proxy is not passed to its constructor.

The micrometer doc says
management.metrics.export.datadog.uri=https://app.datadoghq.com # URI to ship metrics to. If you need to publish metrics to an internal proxy en-route to Datadog, you can define the location of the proxy with this.
But I dont understand what it means? should I replace this url with my proxy url or is there any specific uri pattern with the proxy?

This is referring to a different kind of proxy that you would configure to receive your Datadog traffic on your internal network, and it would forward it to Datadog outside of your network. If you are using an HTTP proxy then you should use the system properties or an HttpSender with your HTTP proxy configured (e.g. an HttpUrlConnectionSender and passing a Proxy to its constructor).

You can configure a custom HttpSender with a DatadogMeterRegistry using its Builder. If you expose this as a Bean in a @Configuration class, Spring Boot will use it in its auto-configuration. For example:

@Bean
public DatadogMeterRegistry datadogMeterRegistry(DatadogConfig config, Clock clock) {
    HttpSender httpSender = new HttpUrlConnectionSender(config.connectTimeout(), config.readTimeout(), new Proxy(Proxy.Type.HTTP, new InetSocketAddress("myproxy", 8080)));
    return DatadogMeterRegistry.builder(config).clock(clock).httpClient(httpSender).build();
}

[1] https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html

Thanks. Tommy, we did set the https proxy details but we had no luck. We will try the spring bean option. Thanks for the help. I will come back and post my result here. Thanks — GAK, Apr 16 '20 at 13:20

Spring boot micro meter datadog socket connection error

1 Answers1