34

In my company, we are using Rythm due to its facility and easy use in a project. In our project, we are sending several e-mails (1000-2000 emails by day); the e-mail template is a Rythm template with dynamic syntax (Java code). Performance seems fine and it passed the integration tests.

Nevertheless, we have experimented several memory problems that drives to a memory leak after 3-4 days. Profiling, we have observed that Rythm is the most biggest object of the heap (our profilings are about 1 day) even more than the ClassLoader or the BeanFactory from Spring.

Using heap tools analyzer, we have observed that RythmEngine and TemplateClassManager are the biggest objects

 (Instance) - (retained size bytes)

org.rythmengine.RythmEngine#1 - 10,192,894
org.rythmengine.internal.compiler.TemplateClassManager#1 - 9,223,065
org.springframework.boot.loader.LaunchedURLClassLoader#1 - 6,975,661
java.util.Vector#89 - 6,378,290
java.lang.Object[]#7549 - 6,378,254
org.springframework.beans.factory.support.DefaultListableBeanFactory#1 - 3,741,643

......

We can see from the heap analyzer tools that these objects are big ones, and it seems that they increase during time.

enter image description here

And a GC Root.

enter image description here

Concerning memory pools: Par Eden seems fine and CMS Old Generation seems not to increase, or at least slowly (even after some major GCs it seems that free memory). Heap memory seems fine (tests and profiling are about one day), but in production increases slowly after it reaches the maximum heap.

We are asking if someone has experimented this feature (using rythm and after some days a memory leak appears) or just give some best practices of how to improve performance with rythm in a production environment. Or any idea of how to deal with a depth memory leak will be welcomed.

IMPORTANT NOTE [30-09-2015] : We have changed from Rythm to FreeMarker as Template Engine and it seems (as our Monitoring Systems reflects) that memory is more stable and it is about of 20% of max memory (-Xmx1024). We will inform more detailed during this week. But it seems that Rythm might have some memory problems that it drives to a memory leak after a couple of days.

IMPORTANT NOTE [06-10-2015] : After some days of intensive monitoring we have checked that memory is stable using FreeMarker as Template Engine. We have removed all the dependencies of Rythm in our product because as our studies reflects it has a potential memory leak problem not solved which drives to a OOME for heap after some days (in our case two days). Issue closed.

Francesc Xavier
  • 341
  • 2
  • 5
  • 2
    Just trying to clarify what it happened, we have found this issue (https://github.com/greenlaw110/Rythm/issues/199) of github. It was demostrated that a memory leak was due to org.rythmengine.DefaultShutdownService implementation (it registered a shutdown hook). But, it was commented this component. It is quite strange, because profiling only shows that these objects are big in retained size and it seems that increases during time (but slowly). Several doubts and not much documentation. – Francesc Xavier Aug 13 '15 at 16:47
  • 1
    I have Rythm running in a Banking project for months and no memory issue found. Is your RythmEngine instance run in PROD mode or DEV mode? An interesting thing is where are those aspectj stuff come from? Rythm doesn't use them? BTW, can you please upload your memory dump file to somewhere so that I can have close look into the issue? Disclaim: I am the author of Rythm engine – Gelin Luo Aug 13 '15 at 21:31
  • 1
    The other question I want to ask is how many Rythm instances do you have in your JVM? – Gelin Luo Aug 13 '15 at 22:06
  • 1
    Thanks for your comment!! We have in prod mode. AspectJ is used in order to deal with TX support using Spring. AspectJ does not present any problem. We are using AspectJ in other projects and it rocks! – Francesc Xavier Aug 14 '15 at 08:51
  • 1
    We have only one instance for RythmEngine and TemplateClassManager. I will try to give you a heap dump file during today or monday. Thanks. – Francesc Xavier Aug 14 '15 at 08:52
  • You can find here a hprof file of a heap dump https://drive.google.com/open?id=0B720uXFUmVgAcGg0eERQRWM3Rlk I will remove the link on monday. Thanks!! – Francesc Xavier Aug 14 '15 at 17:22
  • I can't find anything special from the hprof file. The rythm engine instance is the biggest object as per retained size (about 15M), because it has reference to the second biggest object TempmlateClassManager (about 13M) which refers to the 6th largest object LaunchedURLClassLoader (about 7M) – Gelin Luo Aug 16 '15 at 05:08
  • But do you have observed the same in your production environment?? is it usual? – Francesc Xavier Aug 16 '15 at 14:17
  • No I have Rythm running in several projects some one them going for years. Haven't got that kind of problem. Note `TemplateClassManager` looks big in tools like visualvm because it's class loader refers to your main application's class loader. Also I actually don't see any issue on the hprof file you sent out. I think the memory is okay. Do you have any histogram on memory status over the time period? – Gelin Luo Aug 17 '15 at 03:10
  • yes, we have but it seems correct, I will show you (read before about memory pools) during this morning. thanks – Francesc Xavier Aug 18 '15 at 08:32
  • We will check carefully the size of the object in production environment.... all seems correct and memory pools (and heap memory too) behaves as expected. Thanks for your support, we keep in touch – Francesc Xavier Aug 19 '15 at 09:37
  • Good to know that. I am glad that Rythm helps :-) BTW can you share your system URL? Just want know who is using rythm ;-） – Gelin Luo Aug 20 '15 at 11:14
  • We have performed a refactoring of render engine just changing Rythm for FreeMarker in order to know what it is happening. During this week and the other, we will perform intensive test in order to determine what it is happening. The situation right now, it is that after two, or even less, dayswe notice problems of memory. We have done several things in order to determine what it is happening (findBugs, profiling and hprof) and our code is correct. We will inform in this thread in the following 15 days. – Francesc Xavier Sep 21 '15 at 17:03
  • After one day of monitoring, New Relic and our internal monitoring confirms that memory seems stable using FreeMarker. It seems that Rythm was the cause of the Memory Leak and the OOME as the profiling, heapdump and more claims. We will do some more monitoring in order to confirm such hypothesis. Moreover, we briefly analyze your code of github and FindBugs reflected more than 50 errors, have you checked your code? It is a really pity because your project looks great and first benchmarks reflected a good performance. We will inform more during this week. Thanks – Francesc Xavier Sep 30 '15 at 13:09
  • I really appreciate that. Please keep me updated with your new findings on the memory issue of Rythm. I will follow up with FindBugs ... thanks a lot! – Gelin Luo Oct 01 '15 at 00:32
  • Hey again, memory now is stable using FreeMarker as our monitoring system reveals. Honestly, we do not know what it is happening but it seemed that memory increased heavily (we were rendering more than 2000 emails at day using a template engine Rythm) and after 2-3 days an alert of memory usage close to 100% it was reached. We are quite accurate with our products and we use several tools (such as PMD, findbugs, ...) and it was really annoying for us. Rythm was selected because was easy and cool to use but it seems that have memory problems quite difficult to analyze. Many thanks for your help. – Francesc Xavier Oct 01 '15 at 10:03
  • Now, we are rendering 2000-3000 emails at day and memory is stable around 16-20% of max memory (1024MB of -Xmx). We have performed heapdump analysis, profiling and a lot of features (we have expend a lot of time.. and money) and we only appreciate that these objects are candidates to memory leaks and the biggest objects of the stack. We are going to change Rythm in other modules less critical in favor of FreeMarker. Nevertheless, if you find something, we will appreciate that you communicate us. Thanks. – Francesc Xavier Oct 01 '15 at 10:09
  • Hi thanks for all your effort spending on profiling Rythm. I plan to start working on Rythm memory issue in short time, literally I will try to keep running rythm to generate templates and monitor the memory usage. Is it possible for you to send me a mock of the email templates and template argument models similar to your case? – Gelin Luo Oct 02 '15 at 00:55
  • Good morning, we can not send u template because it is a closed product. However, it has only a loop-for, some variables and some simple snippet code for retrieving current day, current month and so on from java.util.Calendar. A very simple java class resulting. We have checked that maven version available in public repository does not correct the issue with the shutdown hook. The Git version is a snapshot. As we told you we recommend to use some tools like FindBugs to debug errors or bad practices (dead code, unused methods,...) or potential problems in production environments. – Francesc Xavier Oct 02 '15 at 08:14
  • I'm voting to close this question as off-topic because it's a bug report. – dimo414 Apr 30 '16 at 02:21

1 Answers1

2

We have as well faced and came across such issue but its because of compilation of templates again and again. To avoid this we did the below settings, Enable prod mode Enable template caching Set the template compiled directory location - to store the compiled version of template files. (Dont confuse with template directory configuration) which will boost your application speed.

Map<String, Object> rythmConfigs = new HashMap<>();
        //rythmConfigs.put(ENGINE_MODE, this.appMode ? "prod" : "dev");
        rythmConfigs.put(PRECOMPILE_MODE_ENABLED, this.config.getBoolean(PRECOMPILE_MODE_ENABLED, true));
        rythmConfigs.put(LOAD_PRECOMPILED_ENABLED, this.config.getBoolean(LOAD_PRECOMPILED_ENABLED, true));
        rythmConfigs.put("rythm.default.cache_ttl", Integer.MAX_VALUE);
        rythmConfigs.put("rythm.cache.enable", this.appMode);
        //rythmConfigs.put("cache.prod_only.enabled", this.appMode);
        rythmConfigs.put(PRECOMPILED_DIR, getTempDir(config.getString(XoAppConfigKeys.APPLICATION_CONTEXT)).getAbsolutePath());
        rythmConfigs.put(TEMPLATE_DIR, this.templateFolderUri.getPath());
        rythmEngine = new RythmEngine(rythmConfigs);
Hakuna Matata
  • 755
  • 3
  • 13