1

We've developed a Node.js Express app which uses Handlebars as a templating engine.
The problem is that it uses a lot of CPU and we can't understand why.
For example, it uses 25% CPU on single thread for a single user request.
The process is ran on a 8 core/16 thread CPU i9 9900K, and with pm2 with maximum instances (so, 16 instances). 2 GBs of RAM are allocated, out of 64.

We use Redis, but not so intensively on the page which is loaded; we use MariaDB with Sequelize but in the requested page there are no queries, everything is in cache.
We also serve our static files from Nginx (pre-minified and bundled).

The only problem I think it could be, is the continuous JSON parsing from Redis results (we parse each Redis result), but I can't disable it to test, because it's needed to get data for the loading of the page.

Node version: v12.16.1
npm: v6.14.5 (with only few issues in audit, not regarding performance)

We also tried load testing using Loader.io and ab command.
Command used for load testing: $ ab -t 50 -n 1000 -c 1000 [link] in external server
CPU with this command goes up to 100% in almost all threads.

From profiling I just can't understand what's the matter of the problem, everything seems to be only "LazyCompiling". I tried everything, and Node V8 profiling isn't helping.

Can someone help us read this V8 profile log and maybe help finding the bottleneck?
Or does someone have alternative profiling techniques?

V8 logs (tested in single thread, but in 16 threads it's the same): https://pastebin.com/9NhyUDZB

Mauro Pirazzoli
  • 21
  • 1
  • What leads you to believe it has something to do with parsing JSON? Are you processing extremely large data sets? Looks like quite a bit of time is spent on garbage collection (`8034 38.4% 458.0% GC`) which seems to indicate you are running into your 2GB memory limit. – Jake Holzinger May 22 '20 at 21:36
  • @JakeHolzinger I don't think they are very large datasets - only two of those are big: are JSON parsed MariaDB tables - one of 3000 rows and one of 27000. I'm now trying to increase RAM and test again. **Edit:** I tried and and it doesn't change nothing even when I allocated 16 GB of RAM. – Mauro Pirazzoli May 22 '20 at 21:54
  • Interesting, is the garbage collector still being reported as a significant figure when profiling? 27000 rows is not really that large assuming the rows are not data dense. Have you ruled out handlebars? – Jake Holzinger May 22 '20 at 22:14
  • @JakeHolzinger Yeah, it looks that garbage collector is still high (100%+ even if not 400% like previous profiling); I can't understand what do you mean by "ruling out" handlebars? – Mauro Pirazzoli May 22 '20 at 22:27
  • Handlebars is synchronous, it will block the thread as it compiles the template and as it inserts the data into the template, it may be useful to ensure the template is compiled only once and that inserting the data into the template is not where you are taking this performance hit. How large is the HTML page that is being generated? – Jake Holzinger May 22 '20 at 22:32
  • The page is really large sadly, even Google Lighthouse says the page has too many HTML tags. I thought about Handlebars before, but I don't know if you can make Handlebars asynchronous or pre-compile templates; from what I know, tho, express-hbs keeps in cache what it can. – Mauro Pirazzoli May 22 '20 at 22:41
  • I don't think handlebars supports asynchronous rendering, or if the express view engine is smart enough to cache the templates it compiles. If the HTML is really that large switching templating engines may not do much, I would personally consider implementing pagination instead. – Jake Holzinger May 22 '20 at 22:47
  • @JakeHolzinger It's not **that** large - I mean, it surely is above average, but it's only an homepage and don't think it requires pagination, it's just a big document but can't be paginated and it shouldn't; also, could Handlebars really affect this much the entire CPU performance? Could it be the only reason? – Mauro Pirazzoli May 22 '20 at 23:02
  • I can only speculate, if there was an example I could use to reproduce your issue then I may have other insights. It could be anything, looking into Handlebars was merely a suggestion. If you could change the code to simply return the JSON instead of rendering the HTML page I think it would be obvious whether or not Handlebars is the problem. It's hard to provide a solution without first identifying what exactly is causing the performance issue. Try to isolate the problem down to a single component in your ecosystem and provide some code if you can. – Jake Holzinger May 22 '20 at 23:23
  • @JakeHolzinger that's a nice idea. I'll try first to load only JSON without rendering Handlebars; if nothing changes, I'll provide some code snippets. I'll let you know very soon, thanks for helping until now ^^ – Mauro Pirazzoli May 22 '20 at 23:26
  • @JakeHolzinger I tested and it seems that Handlebars is not the problem. It looks like that, even when rendering only the page's JSON, CPU goes high anyway. Probably data fetching is the problem. I'll move this discussion to chat if that's okay for you, that would be nice to fix this problem which we have from a lot of time and I think you're the one that helped more till now, thanks. – Mauro Pirazzoli May 22 '20 at 23:30
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/214455/discussion-between-jake-holzinger-and-mauro-pirazzoli). – Jake Holzinger May 22 '20 at 23:33

0 Answers0