It probably uses a JVM API called JVMTI, see https://en.wikipedia.org/wiki/Java_Virtual_Machine_Tools_Interface, this allows to inject agent-code into a started Java application which inspects the byte code and can also modify byte code for doing it's work.
This requires native code as the interface is below the actual execution of the Java class.
There is also a java agent, see here and here for some description. This is used by other monitoring tools, but using a native agent allows to read and control more aspects of the application in respect to reading monitoring and performance information, which explains some of the more powerful things that Dynatrace can do.