Spring sleuth Runtime Sampling and Tracing Decision

Question

I am trying to integrate my Application with Spring sleuth. I am able to do a successfull integration and I can see spans getting exported to Zipkin.
I am exporting zipkin over http.

Spring boot version - 1.5.10.RELEASE
Sleuth - 1.3.2.RELEASE
Cloud- Edgware.SR2

But now I need to do this in a more controlled way as application is already running in production and people are scared about the overhead which sleuth can have by adding @NewSpan on the methods.

1. I need to decide on runtime wether the Trace should be added or not (Not talking about exporting). Like for actuator trace is not getting added at all. I assume this will have no overhead on the application. Putting X-B3-Sampled = 0 is not exporting but adding tracing information. Something like skipPattern property but at runtime.

2. Always export the trace if service exceeds a certain threshold or in case of Exception.

3. If I am not exporting Spans to zipkin then will there be any overhead by tracing information?

What about this solution? I guess this will work in sampling specific request at runtime.

    @Bean
    public Sampler customSampler(){
    return new Sampler() {
        @Override
        public boolean isSampled(Span span) {
            logger.info("Inside sampling "+span.getTraceId());
            HttpServletRequest httpServletRequest=HttpUtils.getRequest();
            if(httpServletRequest!=null && httpServletRequest.getServletPath().startsWith("/test")){
                return true;
            }else
                return false;

        }
    };
}

Answer 1

people are scared about the overhead which sleuth can have by adding @NewSpan on the methods.

Do they have any information about the overhead? Have they turned it on and the application started to lag significantly? What are they scared of? Is this a high-frequency trading application that you're doing where every microsecond counts?

I need to decide on runtime whether the Trace should be added or not (Not talking about exporting). Like for actuator trace is not getting added at all. I assume this will have no overhead on the application. Putting X-B3-Sampled = 0 is not exporting but adding tracing information. Something like skipPattern property but at runtime.

I don't think that's possible. The instrumentation is set up by adding interceptors, aspects etc. They are started upon application initialization.

Always export the trace if service exceeds a certain threshold or in case of Exception.

With the new Brave tracer instrumentation (Sleuth 2.0.0) you will be able to do it in a much easier way. Prior to this version you would have to implement your own version of a SpanReporter that verifies the tags (if it contains an error tag), and if that's the case send it to zipkin, otherwise not.

If I am not exporting Spans to zipkin then will there be any overhead by tracing information?

Yes, there is cause you need to pass tracing data. However, the overhead is small.