Micrometer integration for spring reactive

Question

I am trying to integrate micrometer with zipkin2 with brave for spring-webflux 3.0.5, below are my gradle dependencies. Here is my GitHub repo for the same

implementation 'io.micrometer:micrometer-core:1.11.0'
implementation( group: 'org.apache.commons', name: 'commons-lang3', version: '3.12.0')
implementation 'io.zipkin.reporter2:zipkin-reporter-metrics-micrometer:2.16.4'
// https://mvnrepository.com/artifact/io.micrometer/micrometer-tracing-bridge-brave
implementation 'io.micrometer:micrometer-tracing-bridge-brave:1.1.1'
// https://mvnrepository.com/artifact/io.zipkin.reporter2/zipkin-reporter-brave
implementation 'io.zipkin.reporter2:zipkin-reporter-brave:2.16.4'

below is my Configuration class

@Configuration(proxyBeanMethods = false)
@Slf4j
public class MicroMeterConfig {


    @Bean
    public ObservedAspect observedAspect(ObservationRegistry observationRegistry) {
        return new ObservedAspect(observationRegistry);
    }

    @Bean
    public MeterRegistry meterRegistry() {
        MeterRegistry meterRegistry = Metrics.globalRegistry;
        log.info("MDC get value:{}", MDC.get("TraceId"));
        meterRegistry.config().commonTags("TraceId", MDC.get("TraceId"));
        return meterRegistry;
    }

}

and below is my TraceIdFilter as I want to keep track of the trace id (or generate a new one if it is not present)

public class TraceIDFilter implements WebFilter {
    public static final String TRACE_ID_HEADER_NAME = "TraceId";
    private static final String STATUS_URI_PATTERN = "_status";

    private final Timer timer;
    private static final Logger logger = Loggers.getLogger(TraceIDFilter.class);

    private final CustomTraceBaggage customTraceBaggage;

    public TraceIDFilter(MeterRegistry meterRegistry, CustomTraceBaggage customTraceBaggage) {
        this.customTraceBaggage = customTraceBaggage;
        this.timer = Timer.builder("trace_id_filter_timer")
                .description("Timer for Trace ID Filter")
                .register(meterRegistry);
    }

    @Override
    public Mono<Void> filter(ServerWebExchange serverWebExchange, WebFilterChain chain) {
        Timer.Sample sample = Timer.start();
        return Mono.just(serverWebExchange)
                .doOnNext(exchange -> {
                    final ServerHttpRequest request = exchange.getRequest();
                    final String uriPath = request.getURI().getPath();
                    final String incomingTraceId = request.getHeaders().getFirst(TRACE_ID_HEADER_NAME);
                    final String traceId = Optional.ofNullable(incomingTraceId).orElse(UUID.randomUUID().toString());
                    customTraceBaggage.updateTraceId(traceId);
                    MDC.put(TRACE_ID_HEADER_NAME, traceId);
                    logConditionally(uriPath, "incomingTraceId={}, HttpMethod={}",
                            incomingTraceId,
                 
                            request.getMethod());
                })
                .doOnNext(exchange -> exchange.getResponse()
                        .beforeCommit(() -> {
                            exchange
                                    .getResponse()
                                    .getHeaders()
                                    .add(TRACE_ID_HEADER_NAME, customTraceBaggage.getTraceId());

                            final ServerHttpRequest request = exchange.getRequest();
                            final String requestUri = request.getURI().getPath();
                            long duration = sample.stop(timer);
                            logConditionally(requestUri, "MSG=Response, incomingTraceId={},  HttpMethod={}, responseStatus={}, content-length={}, timeTaken={}",
                                    (Object) request.getHeaders().getFirst(TRACE_ID_HEADER_NAME),
                                 
                                    request.getMethod(),
                                    exchange.getResponse().getStatusCode(),
                                    exchange.getResponse().getHeaders().getFirst("content-length"), duration);
                            return Mono.empty();
                        })
                )
                .then(chain.filter(serverWebExchange));
    }
    private void logConditionally(final String requestUri, final String format, final Object... arguments) {
       
            logger.info(format, arguments);
        
    }
}

below is MyCustomBaggage configuration class

@Component
@RequiredArgsConstructor
public class CustomTraceBaggage {
    private final BaggageField traceIdBaggage;

    public boolean updateTraceId(final String value) {
        return traceIdBaggage.updateValue(value);
    }

    public String getTraceId() {
        return traceIdBaggage.getValue();
    }
}

but I am getting NPE while registering MicroMeterRegistry as in MDC I am not having TraceId value.

at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
    ... 98 common frames omitted
Caused by: java.lang.NullPointerException: null
Caused by: java.lang.NullPointerException: null

    at java.base/java.util.Objects.requireNonNull(Objects.java:233)
    at io.micrometer.core.instrument.ImmutableTag.<init>(ImmutableTag.java:37)
    at io.micrometer.core.instrument.Tag.of(Tag.java:31)
    at io.micrometer.core.instrument.Tags.of(Tags.java:265)
    at io.micrometer.core.instrument.MeterRegistry$Config.commonTags(MeterRegistry.java:756)
    at com.configuration.MicroMeterConfig.meterRegistry(MicroMeterConfig.java:26)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:578)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:139)
    ... 99 common frames omitted

if anyone point me to the correct source on how I can configure Micrometer with zipkin2 would be great

Edit1:

I updated my Github repo by removing MDC tags and adding netty-core for MacOS, now app started correctly but when I am hitting an endpoint, getting the below error

2023-06-06T12:46:32.925+02:00 ERROR [,,] 13151 --- [ender@5b03a20a}] i.n.r.d.DnsServerAddressStreamProviders  : Unable to load io.netty.resolver.dns.macos.MacOSDnsServerAddressStreamProvider, fallback to system defaults. This may result in incorrect DNS resolutions on MacOS. Check whether you have a dependency on 'io.netty:netty-resolver-dns-native-macos'. Use DEBUG level to see the full stack: java.lang.UnsatisfiedLinkError: failed to load the required native library
2023-06-06T12:46:32.999+02:00  WARN [,,] 13151 --- [ender@5b03a20a}] z.r.AsyncReporter$BoundedAsyncReporter   : Spans were dropped due to exceptions. All subsequent errors will be logged at FINE level.
2023-06-06T12:46:32.999+02:00  WARN [,,] 13151 --- [ender@5b03a20a}] z.r.AsyncReporter$BoundedAsyncReporter   : Dropped 1 spans due to WebClientRequestException(Connection refused: localhost/[0:0:0:0:0:0:0:1]:9411)

org.springframework.web.reactive.function.client.WebClientRequestException: Connection refused: localhost/[0:0:0:0:0:0:0:1]:9411
        at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onNext(FluxConcatMapNoPrefetch.java:211) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.SinkManyEmitterProcessor.drain(SinkManyEmitterProcessor.java:471) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.SinkManyEmitterProcessor$EmitterInner.drainParent(SinkManyEmitterProcessor.java:615) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onError(DefaultPooledConnectionProvider.java:162) ~[reactor-netty-core-1.1.5.jar:1.1.5]
        at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:475) ~[reactor-netty-core-1.1.5.jar:1.1.5]
        at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.lambda$drainLoop$9(SimpleDequePool.java:429) ~[reactor-netty-core-1.1.5.jar:1.1.5]
        at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.maybeOnError(FluxConcatMapNoPrefetch.java:326) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onNext(FluxConcatMapNoPrefetch.java:211) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.SinkManyEmitterProcessor.drain(SinkManyEmitterProcessor.java:471) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.SinkManyEmitterProcessor$EmitterInner.drainParent(SinkManyEmitterProcessor.java:615) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.FluxPublish$PubSubInner.request(FluxPublish.java:602) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:336) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.Operators$DeferredSubscription.request(Operators.java:1717) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:192) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241) ~[reactor-core-3.5.4.jar:3.5.4]
        
    Suppressed: java.lang.Exception: #block terminated with an error
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99) ~[reactor-core-3.5.4.jar:3.5.4]
        at reactor.core.publisher.Mono.block(Mono.java:1710) ~[reactor-core-3.5.4.jar:3.5.4]
        at 
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/[0:0:0:0:0:0:0:1]:9411
Caused by: java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[na:na]
    at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[na:na]
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[na:na]
    at 
    java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/[0:0:0:0:0:0:0:1]:9411

Caused by: java.net.ConnectException: Connection refused

based on some posts where they show how to use spring boot, micrometer and zipkin together (like [Micrometer and Zipkin: How to Trace HTTP Requests in Spring Boot 3](https://www.appsdeveloperblog.com/micrometer-and-zipkin-in-spring-boot/#Adding_Micrometer_to_Spring_Boot_3_Application), wrote a few days ago), you have to add [spring-boot-actuator](https://github.com/spring-projects/spring-boot/tree/v3.0.5/spring-boot-project/spring-boot-actuator) as a dependency to be able to monitor your services — Gastón Schabas, Jun 03 '23 at 23:46
@GastónSchabas yes I have added it. But missed to add in my question — LowCool, Jun 04 '23 at 07:26
could you edit your question and provide a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example)? — Gastón Schabas, Jun 05 '23 at 03:43
I can't find the `micrometer-observation` in the [build.gradle](https://github.com/vihang16/micrometerExample/blob/master/build.gradle). Maybe it is not required. Are you following some tutorial saying that you should use those dependencies? In the one I sahred is using maven and it doesn't set the version of each dependency. Not sure how it work in gradle — Gastón Schabas, Jun 05 '23 at 15:27
@GastónSchabas in your case how it decide which version pick? I am following the tutorial you gave let me try `observation` in my gradle — LowCool, Jun 05 '23 at 17:08

Jonatan Ivanov · Answer 1 · 2023-06-08T02:42:17.323

Please do not define versions and let the Boot BOM define them for you. This is what you need:

implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'io.micrometer:micrometer-tracing-bridge-brave'
implementation 'io.zipkin.reporter2:zipkin-reporter-brave'

You can find a bunck of working samples in our micrometer-samples repo, for example: micrometer-samples-boot3-web. Also, we publish our artifacts to Maven Central, use mvnrepository at your own risk.

There are a few other things that are wrong here:

Boot creates a MeterRegistry for you, you don't need to and you should not use the global registry (Metrics.globalRegistry) in Boot.
You are trying to add high cardinality tags to your metrics: meterRegistry.config().commonTags("TraceId", MDC.get("TraceId")); please read this article to see why this is wrong (you are trying to kill your app there).
I think what you are trying to do is attaching the traceId for every request but your code will only run bean initialization time once, where you will not even have a TrceId, hence the NPE: at com.configuration.MicroMeterConfig.meterRegistry(MicroMeterConfig.java:26)

If you want tracing information (TraceId, SpanId) attached to you metrics, please consider using exemplars, we support them for Prometheus and it works out of the box.

From what I see in the web flux example, you are generating traceId inside the actual call, I need to segregate that as AOP, is it possible to configure it? — LowCool, Jun 09 '23 at 10:11
I updated my code based on your comment, but how do I make sure `traceId` part it does not come from a request, I will generate my own. — LowCool, Jun 09 '23 at 11:44

Micrometer integration for spring reactive

1 Answers1