1

I'm using the Elastic APM Agent as a Java Agent to monitor usages of various methods in my spring boot microservice. This all works fine and we're able to graph various metrics in Kibana. Unfortunately what it doesn't do is consistently attach the same labels to all spans within a transaction e.g. details of the user who made the original request.

To work around this I thought I could use ByteBuddy (which I've never used before) to wrap any usages of the APM Span class and attach that information (as it's readily available from ThreadLocal) to each instance. I am however having issues accessing the Span class as it's in the APM Java Agent and with the following code I get the following logs where it appears that it's not able to find the Span class...

        Instrumentation instrumentation = ByteBuddyAgent.install();
        new AgentBuilder.Default()
                .with(debuggingListener)
                .ignore(ElementMatchers.nameStartsWith("net.bytebuddy."))
                .type(ElementMatchers.named("co.elastic.apm.agent.impl.transaction.Span"), ElementMatchers.isBootstrapClassLoader())
                .transform((builder, type, classLoader, module) -> builder.visit(Advice.to(SpanAdvice.class).on(ElementMatchers.hasMethodName("start"))))
                .installOn(instrumentation);

[Byte Buddy] DISCOVERY co.elastic.apm.agent.impl.transaction.Transaction [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] IGNORE co.elastic.apm.agent.impl.transaction.Transaction [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] COMPLETE co.elastic.apm.agent.impl.transaction.Transaction [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] DISCOVERY co.elastic.apm.agent.impl.transaction.Transaction$1 [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] IGNORE co.elastic.apm.agent.impl.transaction.Transaction$1 [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] COMPLETE co.elastic.apm.agent.impl.transaction.Transaction$1 [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] DISCOVERY co.elastic.apm.agent.impl.transaction.AbstractSpan$ChildDurationTimer [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] IGNORE co.elastic.apm.agent.impl.transaction.AbstractSpan$ChildDurationTimer [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] COMPLETE co.elastic.apm.agent.impl.transaction.AbstractSpan$ChildDurationTimer [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] DISCOVERY co.elastic.apm.agent.impl.transaction.SpanCount [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] IGNORE co.elastic.apm.agent.impl.transaction.SpanCount [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] COMPLETE co.elastic.apm.agent.impl.transaction.SpanCount [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] DISCOVERY co.elastic.apm.agent.impl.transaction.TraceState$TextTracestateAppender [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] IGNORE co.elastic.apm.agent.impl.transaction.TraceState$TextTracestateAppender [null, unnamed module @758a34ce, loaded=false]
[Byte Buddy] COMPLETE co.elastic.apm.agent.impl.transaction.TraceState$TextTracestateAppender [null, unnamed module @758a34ce, loaded=false]

I've tried using ByteBuddy for my own classes and it all works without issue, but I'm getting very confused around which classloader has loaded what and how to point ByteBuddy at them.

ben
  • 31
  • 1
  • Generally, agent code is nothing but code on the class path, there are no actual restrictions in instrumenting this code. Of course, an agent is a solution where another approach is not possible or overly complex, as pointed out. – Rafael Winterhalter Jan 14 '21 at 20:00

2 Answers2

0

If the objective is to attach labels to a transaction over multiple spans then using the public APIs from the Elastic APM for Java is a better choice instead of instrumenting the JVM with ByteBuddy. You will have much more freedom to do what you want to do without relying on a hacking. FYI, the Elastic APM agent for Java already instrument the JVM with additional bytecode so what you are doing may get even more confusing because of this.

Alternatively, you can also use the OpenTracing Bridge to set labels in a transaction.

Ricardo Ferreira
  • 1,236
  • 3
  • 6
  • Wouldn't that mean I'd have to apply the same labels manually everytime a new span was created? Or am I misunderstanding? What I'd like to do is add the same labels to every span under a transaction no matter if the span was created by me or the APM Agent. – ben Jan 13 '21 at 19:53
  • Not necessarily. In your example you said "user who made the original request" which sounds like a custom label to me. Everything else will be added automatically by the APM agent for any supported instrumentation library. Custom labels have to be manually added and you can do this either by using the Elastic APM APIs or using OpenTracing APIs. – Ricardo Ferreira Jan 14 '21 at 14:58
  • So one example of this would be a user calls my microservice, at which point I can add various custom labels to the apm transaction detailing information about the user. My service then goes on to make an HTTP request to another microservice, which has a span created for it by the elastic apm agent. I would like my custom user labels to also be applied to the HTTP span to provide better context on who the call is actually being made on behalf of. My problem is that I can't access the span created by the elastic apm agent, so I can't apply the labels. – ben Jan 15 '21 at 14:22
  • It seems that you are then having problems with the APM agent per-se, which is not collecting the spans from your app as it should be. So let's take one step back and solve this problem because once it is solved you must be able to add custom labels to your existing transaction without much struggle. Did you mark which packages from your Spring Boot microservice should be captured? For your convenience here is the link to the configuration options you need to use in your application: https://www.elastic.co/guide/en/apm/agent/java/current/configuration.html – Ricardo Ferreira Jan 15 '21 at 15:33
  • Thanks for your help with this Ricardo. I have the environment property ELASTIC_APM_TRACE_METHODS set up to trace all Spring method use with "public @org.springframework.stereotype.*". The agent also picks up use of the RestTemplate class as we call other microservices, although this is not something we have specifically configured. – ben Jan 15 '21 at 17:02
  • Got it. RestTemplate is being picked because it falls under the list of automatic instrumented libraries, along with Kafka, Redis, JDBC, etc. I would recommend using `application_packages` instead of specifying each method to trace. Certain methods may not be executed under a propagated context. If you need an example that uses Spring Boot I wrote this example here: https://github.com/riferrei/from-zero-to-hero-with-observability/blob/main/backend-java – Ricardo Ferreira Jan 15 '21 at 21:33
0

Disclaimer: This answer is a stub for now.

You should first try to explore a more canonical way of doing things like Ricardo suggested. If for some reason that does not work, then we could explore ways to instrument your agent class - not so much because I think it is a good idea but because it is technically interesting.

Basically, we would have to find out if maybe the class you want to instrument was already loaded before your ByteBuddy agent gets active. Then you would have to use class retransformation rather than redefinition. You would have to make sure the advice you apply can do its job without the need to change the class structure with regard to method signatures and fields.

  .disableClassFormatChanges()
  .with(RETRANSFORMATION)

You would also need to make sure that the advice and ByteBuddy are visible to the other agent's classloader, e.g. by putting both on the boot class path. But let's not get ahead of ourselves. Explore Ricardo's ideas first, please.

kriegaex
  • 63,017
  • 15
  • 111
  • 202