10

My Elixir app is using about 50% of the CPU, but it really should only be using <1%. I'm trying to figure out what is causing the high CPU usage and I'm having some trouble.

In a remote console, I tried

  1. Listing all processes with Process.list
  2. Looking at the process info with Process.info
  3. Sorting the processes by reduction count
  4. Sorting the processes by message queue length

The message queues are all close to 0, but the reduction counts are very high for some processes. The processes with high reduction counts are named

  1. :file_server_2
  2. ReactPhoenix.ReactIo.Pool
  3. :code_server

(1) and (3) are both present in my other apps, so I feel like it must be (2). This is where I'm stuck. How can I go further and figure out why (2) is using so much CPU?

I know that ReactPhoenix uses react-stdio. Looking at top, react-sdtio doesn't use any resources, but the beam does.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        87 53.2  1.2 2822012 99212 ?       Sl   Nov20 580:03 /app/erts-9.1/bin/beam.smp -Bd -- -root /app -progname app/releases/0.0.1/hello.sh -- -home /root -- -noshell -noshell -noinput -boot /app/
root     13873  0.0  0.0   4460   792 ?        Rs   13:54   0:00 /bin/sh -c deps/react_phoenix/node_modules/.bin/react-stdio

I saw in this StackOverflow post that stdin can cause resource issues, but I'm unsure if that applies here. Anyway, any help would be greatly appreciated!

Jesse Shieh
  • 4,660
  • 5
  • 34
  • 49

2 Answers2

18

Did you try etop?

iex(2)> :etop.start

========================================================================================
 nonode@nohost                                                             14:57:45
 Load:  cpu         0               Memory:  total       26754    binary        143
        procs      51                        processes    8462    code         7201
        runq        0                        atom          292    ets           392

Pid            Name or Initial Func    Time    Reds  Memory    MsgQ Current Function
----------------------------------------------------------------------------------------
<0.6.0>        erl_prim_loader          '-'  458002  109280       0 erl_prim_loader:loop
<0.38.0>       code_server              '-'  130576  196984       0 code_server:loop/1  
<0.33.0>       application_controll     '-'   58731  831632       0 gen_server:loop/7   
<0.88.0>       etop_server              '-'   58723  109472       0 etop:data_handler/2 
<0.53.0>       group:server/3           '-'   19364 2917928       0 group:server_loop/3 
<0.61.0>       disk_log:init/2          '-'   16246  318352       0 disk_log:loop/1     
<0.46.0>       file_server_2            '-'    3838   18752       0 gen_server:loop/7   
<0.51.0>       user_drv                 '-'    3720   13832       0 user_drv:server_loop
<0.0.0>        init                     '-'    2559   34440       0 init:loop/1         
<0.37.0>       kernel_sup               '-'    2093   58600       0 gen_server:loop/7   
========================================================================================

http://erlang.org/doc/man/etop.html

Grych
  • 2,861
  • 13
  • 22
  • 4
    What would I do to make it available in `prod`? Attaching to `iex` and running `:etop.start` results in “** (UndefinedFunctionError) function :etop.start/0 is undefined (module :etop is not available)”. – Aleksei Matiushkin Nov 21 '17 at 15:24
  • 2
    According to the doc linked, you can specify the node you wish to connect `etop` to : `etop -node testnode@myhost -setcookie MyCookie` – Kernael Nov 21 '17 at 19:00
  • 1
    @Kernael if this was an answer to my question, the problem is, as specified by the error message, “module `etop` in not available.” I doubt specifying the node would magically bring this module to avail. – Aleksei Matiushkin Nov 22 '17 at 06:03
  • 2
    Surely you should have the script on your machine ? `erl -s etop` – Kernael Nov 22 '17 at 10:31
  • 1
    @mudasobwa does including `runtime_tools` fix this? See https://tkowal.wordpress.com/2016/04/23/observer-in-erlangelixir-release/. – Dogbert Nov 22 '17 at 13:18
  • 1
    @Dogbert I have `runtime_tools` included as `extra_applications`. – Aleksei Matiushkin Nov 22 '17 at 13:54
  • 1
    @mudasobwa Not only `runtime_tools`, also `observer`. – keroro520 Nov 28 '17 at 08:10
  • 1
    why is `cpu` 0 under the load section and `time` empty? I get that locally as well, and what I really want is a sense of cpu usage per process like would get with `top` in linux – mmrobins Feb 27 '19 at 22:51
0

I had the chance to use WombatOAM. For me it was useful in finding bottlenecks. There is a free trial option available. Hope you get lucky.

https://www.erlang-solutions.com/capabilities/wombatoam/

z5ottu
  • 107
  • 5