24

i have app on Pyramid. I run it in uWSGI with these config:

[uwsgi]
socket = mysite:8055
master = true
processes = 4
vacuum = true
lazy-apps = true
gevent = 100

And nginx config:

server {
    listen 8050;
    include uwsgi_params;

    location / {
        uwsgi_pass mysite:8055;
    }
}

Usually all fine, but sometimes uWSGI kills workers. And i have no idea why.

I see in uWSGI logs:

DAMN ! worker 2 (pid: 4247) died, killed by signal 9 :( trying respawn ...
Respawned uWSGI worker 2 (new pid: 4457)

but in the logs there is no Python exceptions.

sometimes i see in uWSGI logs:

invalid request block size: 11484 (max 4096)...skip
[uwsgi-http key: my site:8050 client_addr: 127.0.0.1 client_port: 63367] hr_instance_read(): Connection reset by peer [plugins/http/http.c line 614]

And nginx errors.log:

*13388 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1,
*13955 recv() failed (104: Connection reset by peer) while reading response header from upstream, client:

I think this can be solved by adding buffer-size=32768, but it is unlikely due to this uWSGI kill workers.

Why uwsgi can kill workers? And how can I know the reason? The line "DAMN ! worker 2 (pid: 4247) died, ..." nothing to tells.

Greg Eremeev
  • 1,760
  • 5
  • 23
  • 33

4 Answers4

10

signal 9 means it received a SIGKILL. so something sent a kill to your worker. it's relatively likely that the out-of-memory killer decided to kill your app because it was using too much memory. try watching the workers with a process monitor and see if it uses a lot of memory.

doxin
  • 698
  • 1
  • 7
  • 22
  • 9
    "try watching the workers with a process monitor and see if it uses a lot of memory". How do I do this? – elethan Oct 05 '17 at 18:50
  • 1
    Just to add for someone: Use this uwsgitop https://pypi.python.org/pypi/uwsgitop/0.10 or htop command and filter records to uwsgi and see how much memory is being consumed – Saad Abdullah Mar 19 '18 at 22:46
  • I'm not sure how it's self-obvious from the uwsgitop command to filter records. When I followed those steps it gave an automatically updating screen in the terminal, although, it _did_ include memory when I included the `--memory-report` flag. – AlanSE Nov 15 '18 at 16:20
2

Try to add harakiri-verbose = true option in the uWSGI config.

Mikhail Gerasimov
  • 36,989
  • 16
  • 116
  • 159
0

I had the same problem, for me changing the uwsgi.ini file, changing the value of the reload-on-rss setting from 2048 to 4048, and harakiri to 600 solved the problem.

Alexandre Neukirchen
  • 2,713
  • 7
  • 26
  • 36
0

For me it was that I hadn't filled out app.config["SERVER_NAME"] = "x"

Duarte
  • 109
  • 5