0

I have a problem with my Javamelody Collector server. I start a Tomcat server locally that runs the Javamelody Collector server and I start three applications that register to the Collector server upon starting them. I then send a few request (10 requests or similar) to the applications in order to check if Javamelody recognizes them correctly which it does.

However, when I add an aggregate all where I take the three applications together problems start:

Firstly, the count of http requests of the application "all" is often not equal to the sum of the http requests of the other three applications. Instead, I observed that it was double or equal to the number of http requests of one of the applications. Sometimes, it was also assigned right.

My main problem comes after that though. Javamelody increases the number of hits by itself when I let it run. It does so in the aggregation "all" and in (from what I could reproduce by the steps described in the beginning) one of the applications. At least one time it increased the number of hits in two of the three applications but I was not able to reproduce that.

Moreover, it seems like the number of requests is exponentially growing (roughly doubling every time Javamelody calls the applications to get the information), so that I end up with trilions of requests after a short period of time and such that the number of requests even overflows.

I tried to track the malbehavior down but I cannot solve it. I tracked the addHits function (in CounterRequest.java) and saw that Javamelody adds big numbers of requests there. I could not track the error down with debugging because the places where the addHit method was invoked differed so I couldnt pinpoint the location of the error.

Attached to this I add some pictures of my configuration of the collector server together with the behavior I could observe.

I would be very happy if you could help me out here.

Kind regards, Janis

Overflow of http hits

Behavior of hits(1)

Behavior of hits(2)

Behavior of hits(3)

Configurations

Janis
  • 1
  • 1

1 Answers1

1

There was an issue in javamelody when using an aggregation of monitored application in the optional collect server.

I think that it's now fixed by https://github.com/javamelody/javamelody/commit/f77e5d6 You can download a snapshot build of the collect server including the fix at http://javamelody.org/downloads/javamelody-collector-server-1.87.0-SNAPSHOT.war

Let us know if it is fixed for you.

evernat
  • 1,713
  • 13
  • 18
  • Thank you very much!! The server works way better now! The initialization of the aggregate "all" including all the applications is now correct. Moreover, the hit counts of all the applications stay correct if I let it run for some time. However, there is still a problem in the count of the aggregate. It doesn't grow exponentially anymore but the collector server still adds request hits to the aggregate even though no requests were made or recorded in the three applications. I tried it out with different number of requests sent to the applications several times and noticed a pattern: – Janis Dec 24 '20 at 14:31
  • Say the aggregate "all" initially has n request hits. Then this number n gets added to the hit count of the aggregation "all" every minute. So for example if my three applications have 11, 10 and 20 hits when I add the aggregate "all" it gets initialized with 41 hits. But the hit count after 1, 2, 3 and 4 minutes is 82, 123, 164 and 205 hits. – Janis Dec 24 '20 at 14:32
  • I also noticed that when I stop the collector server and rerun it the bad behavior is gone and the whole collector server works perfectly. Hence, I suspect that the collector server sets a variable to initiallize the number of counts in the aggregate but fails to overwrite it with 0 afterwards. Stopping the server probably deletes this variable or sets it to 0 and therefore leads to the application running correctly afterwards. Another option would be that the variable is meant to be used only for initiallization but is actually used incorrectly at another place in the code. – Janis Dec 24 '20 at 14:32
  • There is one thing which is maybe worth noting as well: If I define the aggregation when all the applications have zero http request count the aggregate gets initialized correctly with zero. However, the moment I do requests on the applications (eg one request for each application) the aggregate gets the http count 3 and then subsquently adds 3 to its count every minute from that moment on. – Janis Dec 24 '20 at 15:08
  • Indeed, there was another issue: that behavior of increasing every minute, if the collector server was not restarted after adding the aggregation. – evernat Dec 26 '20 at 00:35
  • It's fixed by https://github.com/javamelody/javamelody/commit/93ac1b0 You can download again a snapshot build at http://javamelody.org/downloads/javamelody-collector-server-1.87.0-SNAPSHOT.war to check that it's ok for you. – evernat Dec 26 '20 at 00:42
  • 1
    Note for the future: it's great that this quite new feature of aggregations from other applications in the optional collect server is battle tested :-) But if you finally don't mind having the separated graphs and statistics and data storage of each application, you can just use the same application name when registering the monitored applications. That way you would have the reports aggregated for all the (3) started applications, without having to add the aggregation for them. And data storage would be reduced from 3+1 datasets to a single aggregated one. – evernat Dec 26 '20 at 00:47
  • Of course you could also use only the separated monitoring from the 3 applications directly, without the collect server. In that case, you would lose aggregated reports and data storage would be in the servers of the applications. – evernat Dec 26 '20 at 00:53
  • Thank you very very much!! The counter error vanished and the server now works as expected. There are still have some issues but they are probably related to misconfigurations. Thank you also for the further hints on how the collector server can be used. In our use case we relied on the aggregate as we wanted all the statistics :) I tested out some other things and options in the collector server and ran into another problem in that context: – Janis Jan 12 '21 at 13:43
  • If one would want to make the monitoring hierarchical, ie. monitor single threads, organise the different applications as aggreagates of those threads and summarizing all the application (aggregates) in an aggregation "all" the server still reaches its limits. In my case, I tried it out with the three applications: I summarized two of them as an aggreagate A and then added an aggregate "all" as aggreagate of the third (leftover) application together with the aggregate A. – Janis Jan 12 '21 at 13:44
  • Doing what I described above, I got a ConcurrentModificationException. I created a breakpoint for it and I could track it down to the collectWithoutErrors method in CollectorServer.java line 96. The consequence of the exception was that the counts of the aggregates were slightly wrong afterwards and one of them would have a red dot (and the ConcurrentModificationException as info text) exception. – Janis Jan 12 '21 at 13:45
  • The ConcurrentModification for aggregation of aggregation is now fixed by https://github.com/javamelody/javamelody/commit/29089ad. A new snapshot build is available at http://javamelody.org/downloads/javamelody-core-1.87.0-SNAPSHOT.jar Thank you – evernat Jan 20 '21 at 23:21
  • Hi evernat, thanks for helping out once more!! The ConcurrentModificationException does not occur anymore and the server works perfect once its up. However, there is still a minor bug upon startup/innitialization around the aggregates: – Janis Jan 28 '21 at 15:34
  • When I add some applications and build an aggregate out of them then the first aggregate I add to the collection server has double http request count. As for the aggregation that is added second: I dont really know what the pattern is there but its sometimes right and sometimes slightly to high (like when its an aggregate of two applications in would count only one of them double). The third aggregate and all aggregations that are added afterwards are innitialized with the correct hit count. Best regards. – Janis Jan 28 '21 at 15:35