6

I have been following this tutorial to try and get Flask SocketIO running using nginx and gunicorn.

nginx

server {
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_redirect off;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    location /socket.io {
        proxy_pass http://localhost:8000/socket.io;
        proxy_redirect off;
        proxy_buffering off;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";

    }

}

gunicorn_config.py

bind = '127.0.0.1:8000'
workers = 2
worker_class = 'socketio.sgunicorn.GeventSocketIOWorker'

In Supervisor I call my app with:

[program:gunicorn-couponmonk]
directory = ~/couponmonk_project
command =~/venv/py2.7/bin/python    ~/venv/py2.7/bin/gunicorn -c ~/venv/py2.7/lib/python2.7/site-packages/gunicorn/gunicorn_config.py __init__.py 
stdout_logfile = /var/log/gunicorn/couponmonk-std.log
stderr_logfile = /var/log/gunicorn/couponmonk-err.log
user = mint

With this configuration, my app (without using Flask SocketIO) works fine.

I'm just confused at to how to use socketIO. If I go to the address http://localhost:8000/socket.io, I get the response Internal Server Error.

The example HTML/Javascript (https://github.com/miguelgrinberg/Flask-SocketIO/blob/master/example/templates/index.html) contains this line:

var socket = io.connect('http://' + document.domain + ':' + location.port + namespace);

I have two questions:

1) What address is this variable supposed to point to?

2) Is my configuration (nginx, gunicorn) correct?

Sorry if this question is silly. I'm just confused as to how all this is supposed to work.

Thanks for your help.

**UPDATE**

This is the output of the nginx error.log file when I try and access http://localhost/socket.io.

2015/06/20 14:05:08 [debug] 1917#0: *1 http cleanup add: 00000000009F2170
2015/06/20 14:05:08 [debug] 1917#0: *1 get rr peer, try: 1
2015/06/20 14:05:08 [debug] 1917#0: *1 socket 12
2015/06/20 14:05:08 [debug] 1917#0: *1 epoll add connection: fd:12 ev:80000005
2015/06/20 14:05:08 [debug] 1917#0: *1 connect to 127.0.0.1:8000, fd:12 #2
2015/06/20 14:05:08 [debug] 1917#0: *1 http upstream connect: -2
2015/06/20 14:05:08 [debug] 1917#0: *1 posix_memalign: 0000000000A14170:128 @16
2015/06/20 14:05:08 [debug] 1917#0: *1 event timer add: 12: 60000:1434773168437
2015/06/20 14:05:08 [debug] 1917#0: *1 http finalize request: -4, "/socket.io?" a:1, c:2
2015/06/20 14:05:08 [debug] 1917#0: *1 http request count:2 blk:0
2015/06/20 14:05:08 [debug] 1917#0: *1 post event 0000000000A295E0
2015/06/20 14:05:08 [debug] 1917#0: *1 post event 0000000000A29648
2015/06/20 14:05:08 [debug] 1917#0: *1 delete posted event 0000000000A29648
2015/06/20 14:05:08 [debug] 1917#0: *1 http upstream request: "/socket.io?"
2015/06/20 14:05:08 [debug] 1917#0: *1 http upstream send request handler
2015/06/20 14:05:08 [debug] 1917#0: *1 http upstream send request
2015/06/20 14:05:08 [debug] 1917#0: *1 chain writer buf fl:1 s:439
2015/06/20 14:05:08 [debug] 1917#0: *1 chain writer in: 00000000009F21A8
2015/06/20 14:05:08 [debug] 1917#0: *1 writev: 439
2015/06/20 14:05:08 [debug] 1917#0: *1 chain writer out: 0000000000000000
2015/06/20 14:05:08 [debug] 1917#0: *1 event timer del: 12: 1434773168437
2015/06/20 14:05:08 [debug] 1917#0: *1 event timer add: 12: 60000:1434773168437
2015/06/20 14:05:08 [debug] 1917#0: *1 delete posted event 0000000000A295E0
2015/06/20 14:05:08 [debug] 1917#0: *1 http run request: "/socket.io?"
2015/06/20 14:05:08 [debug] 1917#0: *1 http upstream check client, write event:1, "/socket.io"
2015/06/20 14:05:08 [debug] 1917#0: *1 http upstream recv(): -1 (11: Resource temporarily unavailable)
2015/06/20 14:05:08 [debug] 1917#0: *1 post event 0000000000A15E38
2015/06/20 14:05:08 [debug] 1917#0: *1 post event 0000000000A29648
2015/06/20 14:05:08 [debug] 1917#0: *1 delete posted event 0000000000A29648
2015/06/20 14:05:08 [debug] 1917#0: *1 http upstream request: "/socket.io?"
2015/06/20 14:05:08 [debug] 1917#0: *1 http upstream dummy handler
2015/06/20 14:05:08 [debug] 1917#0: *1 delete posted event 0000000000A15E38
2015/06/20 14:05:08 [debug] 1917#0: *1 http upstream request: "/socket.io?"
2015/06/20 14:05:08 [debug] 1917#0: *1 http upstream process header
2015/06/20 14:05:08 [debug] 1917#0: *1 malloc: 00000000009E88A0:4096
2015/06/20 14:05:08 [debug] 1917#0: *1 recv: fd:12 244 of 4096
2015/06/20 14:05:08 [debug] 1917#0: *1 http proxy status 500 "500 Internal Server Error"
2015/06/20 14:05:08 [debug] 1917#0: *1 http proxy header: "Connection: close"
2015/06/20 14:05:08 [debug] 1917#0: *1 http proxy header: "Content-Type: text/html"
2015/06/20 14:05:08 [debug] 1917#0: *1 http proxy header: "Content-Length: 141"
2015/06/20 14:05:08 [debug] 1917#0: *1 http proxy header done
2015/06/20 14:05:08 [debug] 1917#0: *1 xslt filter header
2015/06/20 14:05:08 [debug] 1917#0: *1 HTTP/1.1 500 Internal Server Error
Server: nginx/1.4.6 (Ubuntu)
Date: Sat, 20 Jun 2015 04:05:08 GMT
Content-Type: text/html
Content-Length: 141
Connection: keep-alive

Not sure how helpful this is but I don't know where else to look for extra info.

I'm also curious as to why I can access this: http://localhost/socket.io123abc

and still get an Internal Server Error as opposed to a Not Found error?

I also updated my supervisord.conf file based on Miguel's answer below:

supervisord.conf

[program:gunicorn-couponmonk]
directory = /home/giri/couponmonk_project
command = /home/giri/venv/py2.7/bin/python /home/giri/venv/py2.7/bin/gunicorn --worker-class socketio.sgunicorn.GeventSocketIOWorker __init__:app 
stdout_logfile = /var/log/gunicorn/couponmonk-std.log
stderr_logfile = /var/log/gunicorn/couponmonk-err.log
user = mint

/var/log/gunicorn/couponmonk-err.log

2015-06-20 14:30:11 [3821] [ERROR] Connection in use: ('127.0.0.1', 8000)
2015-06-20 14:30:11 [3821] [ERROR] Retrying in 1 second.
2015-06-20 14:30:12 [3821] [ERROR] Connection in use: ('127.0.0.1', 8000)
2015-06-20 14:30:12 [3821] [ERROR] Retrying in 1 second.
2015-06-20 14:30:13 [3821] [ERROR] Connection in use: ('127.0.0.1', 8000)
2015-06-20 14:30:13 [3821] [ERROR] Retrying in 1 second.

This list continues on for a while..

  • You likely have some python tracebacks in the `/var/log/gunicorn/couponmonk-*` files. Can you find those and post them? – David K. Hess Jun 20 '15 at 04:25
  • @DavidK.Hess I have posted the output. Thanks for your help. –  Jun 20 '15 at 04:33
  • 1
    That looks like you have accidentally started the same process twice. Double-check your process listing to make sure only one instance of the gunicorn master process is running. – David K. Hess Jun 20 '15 at 04:35

3 Answers3

2

I recommend that you make Flask-SocketIO work without nginx and gunicorn. Once you can get it to work through the native gevent server you can move to your real setup.

Regarding your questions:

1) What address is this variable supposed to point to?

Your connection statement is correct. Socket.IO will take the host, port and namespace and build the connection URL on its own, including the /socket.io component. You do not need to specify that in your connection.

2) Is my configuration (nginx, gunicorn) correct?

I think the nginx config is correct. You seem to have copied it direct from my documentation, and I have verified that it works.

The gunicorn config I'm not sure, you are not showing enough of your project to tell. The command that I use, which you should have in your supervisor config, is this:

gunicorn --worker-class socketio.sgunicorn.GeventSocketIOWorker module:app

Where module is the main module of the application, and app is the name of the Flask application instance. You should definitely use a single worker, don't use two workers when using SocketIO.

Miguel Grinberg
  • 65,299
  • 14
  • 133
  • 152
  • Thanks for your help Miguel. Looks like the `internal server error` was related to me running multiple `gunicorn` instances. –  Jun 20 '15 at 05:46
1

First, I'll mention that I don't recommend using SocketIO at all. It adds a little bit of useful functionality over WebSockets but it makes true load balancing with multiple workers (horizontal scaling) impossible unless you make clients sticky to individual workers or use something like Redis to share state information between them. I recommend taking a look at:

https://github.com/youen/gevent-websocket

Native WebSockets is simpler and easier and scales across multiple frontend workers for true load balancing. The code it takes to implement chat room logic is very minimal.

Having said that, here's the answer to your questions:

1) What address is this variable supposed to point to?

Your namespace is going to be "". (According to Miguel, author of Flask-SocketIO, the Javascript client automatically inserts the "socket.io" part.)

So, your io.connect url should be "http://[hostname]/".

2) Is my configuration (nginx, gunicorn) correct?

Ngnix config seems correct (though see next section). If you decide to use WebSockets, you might consider adding proxy_read_timeout 3600;. Otherwise, unless you have a chatty protocol, the WebSocket will be dropped by Nginx once a minute (the default value). (Also, according to Miguel, SocketIO has heartbeats which deal with this.)

Gunicorn config is not correct. With SocketIO you have a couple of choices:

  1. Set workers = 1 in gunicorn so that each SocketIO client is talking to the same worker process.
  2. Change your nginx configuration to use ip-hash command which will cause clients to be assigned to workers by client IP address.
  3. Use Redis or some other database to allow each worker to share state information (though it's not clear anybody has that working yet for Gevent SocketIO).

These choices are forced because SocketIO uses a stateful setup mechanism which breaks when you try to scale horizontally. See this issue for more information: https://github.com/abourget/gevent-socketio/issues/112

And here's a link to SocketIO's documentation talking about it also: http://socket.io/docs/using-multiple-nodes/

If you are getting an Internal Server Error then there is likely an exception being logged somewhere. Try to hunt it down and add it to your question.

Note that you can't test this by hitting that URL in the address bar of your browser – the browser doesn't speak the proper WebSocket protocol by default and will do nothing useful for you. WebSocket connections must be setup using the Javascript API.

Also, trying to hit that URL with the port number bypasses nginx - which is probably not what you want to do. Nginx usually listens at 80/443 and forwards requests to localhost:8000 (that's called a "reverse proxy" setup).

David K. Hess
  • 16,632
  • 2
  • 49
  • 73
  • 2
    There are many inaccuracies in your answer. *It adds a little bit of useful functionality over WebSockets* Socket.IO works over regular HTTP as well as WebSocket, it chooses the best transport for each client. It also keeps track of the connected users and allows the server to broadcast to all uses, or to groups of users that enter a room. With WebSocket none of that exists. *Native WebSockets is simpler and easier and scales across multiple frontend workers without having to deploy something like Redis* Sure, but it's not because of WebSocket, it is because there is no broadcasting/rooms. – Miguel Grinberg Jun 19 '15 at 18:31
  • *Because of how you've got your SocketIO traffic segregated with a separate location in the nginx config, your namespace by definition is /socket.io* This is incorrect, the `/socket.io` resource name is added by the Socket.IO client, it does not go in the connection URL. *Otherwise, unless you have a chatty protocol, the underlying WebSocket will be dropped by Nginx once a minute (the default value).* The heart beat packets in Socket.IO will prevent this. – Miguel Grinberg Jun 19 '15 at 18:33
  • I guess it depends on your perspective. With WebSocket so pervasive now (http://caniuse.com/#feat=websockets) there's no need for alternative transports. Also, the amount of server side logic it takes to implement chat rooms can fit in less than 100 lines of code. I know because I've done it. For me, the deal breaker was needing something like Redis to enable multiple frontend workers. SocketIO doesn't provide nearly enough usefulness to make that worthwhile. I'll have to take your word on the namespace and heartbeat parts and will update my answer. – David K. Hess Jun 19 '15 at 21:35
  • *Also, the amount of server side logic it takes to implement chat rooms can fit in less than 100 lines of code* And how do you plan to broadcast chat messages to all users when you have multiple workers, and without using the shared storage you say you don't like? – Miguel Grinberg Jun 19 '15 at 22:23
  • Same thing would work with Socket.IO. There is absolutely no link between Redis and gevent-socketio. – Miguel Grinberg Jun 19 '15 at 22:42
  • I didn't say there was. However, there is a link between Socket.IO, multiple workers and needing shared storage as documented here: https://github.com/miguelgrinberg/Flask-SocketIO/issues/35 The problem is that the SocketIO protocol has a stateful setup mechanism which doesn't scale horizontally without shared storage – and nobody has gotten it working for Gevent SocketIO yet. – David K. Hess Jun 19 '15 at 22:48
  • Of course it is stateful, it needs to know what users are connected. It is as stateful as your 100 line chat application based on WebSocket + Postgres. You seem to imply being stateful as a disadvantage of the Socket.IO protocol, but it is really a requirement of the applications that need to broadcast messages to groups of users. You can deploy multiple workers with Socket.IO, and as long as you don't use the broadcast/room features you'll be fine. There is really no difference between bare WebSocket and Socket.IO on this respect. – Miguel Grinberg Jun 19 '15 at 22:57
  • There is a subtlety here that I think you might be missing – my point has nothing to do with established sessions – it's about session establishment. The SocketIO protocol has a two step setup process. If the first http request goes to one worker and the second http request goes to a different one the SocketIO connection will not establish. It requires something like Redis for the workers to share the state information between step 1 and step 2 for true load balancing to work. – David K. Hess Jun 19 '15 at 23:05
  • SocketIO's own documentation references the fact that clients must be sticky: http://socket.io/docs/using-multiple-nodes/ That's why you must have either one worker, use nginx ip-hash or use shared storage to share socket session information. – David K. Hess Jun 19 '15 at 23:15
  • All you need to do to use multiple workers with gevent-socketio is to configure your load balancer to forward requests for a client to the same host, that standard thing for stateful services. There are many ways to achieve this (cookies, hashing functions, etc.) without shared storage. Obviously you don't have the ability to broadcast with this setup, but that's true also for bare WebSocket under the same configuration. Your claim in the first and third paragraphs of your answer that Flask-Socket.IO needs "something like Redis" to use multiple workers is incorrect. – Miguel Grinberg Jun 19 '15 at 23:26
  • Fair enough. I updated those first two paragraphs to better coordinate with the later explanation. Actually, front end workers are normally stateless so that they can scale horizontally. I don't think making them stateful is "standard". I would recommend you update your sample nginx configuration to include the "ip-hash" command so that people who happen to use multiple workers don't get bitten by this. – David K. Hess Jun 19 '15 at 23:52
  • @DavidK.Hess Thanks for the explanation. I have updated my question to include the output from the `nginx` `error.log`. –  Jun 20 '15 at 04:19
1

For anyone who is reading this old post, gunicorn --worker-class socketio.sgunicorn.GeventSocketIOWorker module:app command is outdated as Miguel mentioned in one of the github flask-socketio issue.

Simon Lau
  • 61
  • 1
  • 5