22

Is there anything specific that can be done to help make a Django Channels server less susceptible to light or accidental DDoS attack or general load increase from websocket/HTTP clients? Since Channels is not truly asynchronous (still workers behind the scenes), I feel like it would be quite easy to take down a Channels-based website - even with fairly simple hardware. I'm currently building an application on Django Channels and will run some tests later to see how it holds up.

Is there some form of throttling built in to Daphne? Should I implement some application-level throttling? This would still be slow since a worker still handles the throttled request, but the request can be much faster. Is there anything else I can do to attempt to thwart these attacks?

One thought I had was to always ensure there are workers designated for specific channels - that way, if the websocket channel gets overloaded, HTTP will still respond.

Edit: I'm well aware that low-level DDoS protection is an ideal solution, and I understand how DDoS attacks work. What I'm looking for is a solution built in to channels that can help handle an increased load like that. Perhaps the ability for Daphne to scale up a channel and scale down another to compensate, or a throttling method that can reduce the weight per request after a certain point.

I'm looking for a daphne/channels specific answer - general answers about DDoS or general load handling are not what I'm looking for - there are lots of other questions on SO about that.

I could also control throttling based on who's logged in and who is not - a throttle for users who are not logged in could help.

Edit again: Please read the whole question! I am not looking for general DDoS mitigation advice or explanations of low-level approaches. I'm wondering if Daphne has support for something like:

  • Throttling
  • Dynamic worker assignment based on queue size
  • Middleware to provide priority to authenticated requests

Or something of that nature. I am also going to reach out to the Channels community directly on this as SO might not be the best place for this question.

Jamie Counsell
  • 7,730
  • 6
  • 46
  • 81
  • DDOS protection with software firewall ? My opinion is "awe-full" idea ! Read 60 bytes headers of packets on every requests. Triggered more function for reject-drop-freeze is `system resource` issue. Delay is `foregone conclusion`. You may success on low value attacks but how to handle system resources ? Can't use any store procedure on firewall, mean good_user vs bad_user requests. **You can't protect yourself if accepted a unwanted requests !** Not cheap if a packet reached loopback(`lo`). – dsgdfg Jul 21 '17 at 07:50

5 Answers5

6

I've received an answer from Andrew Godwin. He doesn't use StackOverflow so I'm posting it here on his behalf.

Hi Jamie,

At the moment Channels has quite limited support for throttling - it pretty much consists of an adjustable channel size for incoming connections which, when full, will cause the server to return a 503 error. Workers are load-balanced based on availability due to the channels design, so there's no risk of a worker gaining a larger queue than others.

Providing more advanced DoS or DDoS protection is probably not something we can do within the scope of Channels itself, but I'd like to make sure we provide the appropriate hooks. Were there particular things you think we could implement that would help you write some of the things you need?

(It's also worth bearing in mind that right now we're changing the worker/consumer layout substantially as part of a major rewrite, which is going to mean different considerations when scaling, so I don't want to give too precise advice just yet)

Andrew

He's also written about the 2.0 migration in his blog.

Community
  • 1
  • 1
Jamie Counsell
  • 7,730
  • 6
  • 46
  • 81
0

I am only answering the first question. So basically it is impossible to be 100% protected from ddos attacks, because it always comes down to a battle of resources. If the server-side resources are greater than the attacker-side resources, the server will not go down (there may be slowed performance though) but if not, the server goes down [no reference required]. Why is it not possible to be 100% protected, you may ask. So basically your server "crashes" if people cannot connect to it [https://en.wikipedia.org/wiki/Crash_(computing)#Web_server_crashes --- Web server crashes sentence 1.]. So if you try to protect your server by shutting it down for 5 mins every time 10000 connections are made in a second, the ddos succeeded. It "crashed" your server. The only ddos protection that I know of that should work is Cloudfare (https://www.cloudflare.com/lp/ddos-b/?_bt=207028074666&_bk=%2Bddos%20%2Bprotection&_bm=b&_bn=g&gclid=EAIaIQobChMIu5qv4e-Z1QIVlyW9Ch2YGQdiEAAYASAAEgJbQ_D_BwE). It absorbs the impact of the ddos attack with its 10Tbps network backbone. But even it does not offer 100% ddos protection because once its 10Tbps is down, your server will go down too. So, I hope that helped.

Evgeny
  • 153
  • 16
  • Thanks for your answer, @Evgeny, but it doesn't really answer the question. I understand how DDoS attacks work and I understand that you can never be secure. I also know options like Cloudflare exists, but their effectiveness is drastically limited for attacks over websockets because they can't offer user-acceptable mitigation strategies like CAPTCHAs. I'm specifically asking about best practices when building a Django Channels site with a Daphne ASGI server. – Jamie Counsell Jul 21 '17 at 19:13
0

DDoS = Distributed Denial of Service

The 'Distributed' part is the key: you can't know you're being attacked by 'someone' in particular, because requests come from all over the place.

Your server will only accept a certain number of connections. If the attacker manages to create so many connections that nobody else can connect, you're being DDoS'ed.

So, in essence you need to be able to detect that a connection is not legit, or you need to be able to scale up fast to compensate for the limit in number of connections.

Good luck with that!

DDoS protection should really be a service from your cloud provider, at the load balancer level.

Companies like OVH use sophisticated machine learning techniques to detect illegitimate traffic and ban the IPs acting out in quasi-real time. For you to build such a detection machinery is a huge investment that is probably not worth your time (unless your web site is so critical and will lose millions of $$$ if it's down for a bit)

MrE
  • 19,584
  • 12
  • 87
  • 105
  • Hey @MrE, for sure. I totally understand that cloud level protection is better, but I'm also asking about accidental issues with load. Something like throttling that will reduce the amount of impact per request, or something specific to channels that can help, such as allowing Daphne to scale a certain channel down to compensate for another channel. The problem with websockets is they're notoriously hard to protect. I'll update my question to be a bit more clear. – Jamie Counsell Jul 24 '17 at 13:47
  • the amount of traffic in a single websocket really depends on your application, if you expect low traffic (like a chat channel) then yes throttle is a good solution: pause or kick out anybody who is sending too many messages. For other things like IoT data, you should also have an expected rate of data. Throttling in general seems like a good idea. – MrE Jul 24 '17 at 15:49
  • but the best way to handle unknown traffic would be to not process anything with your websocket api server, and rather directly send the messages to a message queue (like Kafka, RabbitMQ) that then can handle spikes and push messages to the process that will actually handle the processing. – MrE Jul 24 '17 at 15:50
  • ... that's what channels does. Again, @MrE I'm really looking for some insight from the Django Channels community here. – Jamie Counsell Jul 24 '17 at 17:37
0

Theres a lot of things you cant to do about DDOS..however there are some neat 'tricks' depending on how much resources you have at your disposal, and how much somebody wants to take you offline.

Are you offering a total public service that requires direct connection to the resource you are trying to protect?

If so, you just going to need to 'soak up' DDOS with the resources you have, by scaling up and out... or even elastic... either way it's going to cost you money!

or make it harder for the attacker to consume your resources. There are number of methods to do this.

If you service requires some kind of authentication, then separate your authentication services from the resource you are trying to protect.

Many applications, the authentication and 'service' run on the same hardware. thats a DOS waiting to happen.

Only let fully authenticated users access the resources you are trying to protect with dynamic firewall filtering rules. If your authenticated then gate to the resources opens (with a restricted QOS in place) ! If your a well known, long term trusted users, then access the resource at full bore.

Have a way of auditing users resource behaviour (network,memory,cpu) , if you see particular accounts using bizarre amounts, ban them, or impose a limit, finally leading to a firewall drop policy of their traffic.

Work with an ISP that can has systems in place that can drop traffic to your specification at the ISP border.... OVH are your best bet. an ISP that exposes filter and traffic dropping as an API, i wish they existed... basically moving you firewall filtering rules to the AS border... niiiiice! (fantasy)

It won't stop DDOS, but will give you a few tools to keep resources wasted a consumption by attackers to a manageable level. DDOS have to turn to your authentication servers... (possible), or compromise many user accounts.... at already authenticated users will still have access :-)

If your DDOS are consuming all your ISP bandwidth, thats a harder problem, move to a larger ISP! or move ISP's... :-). Hide you main resource, allow it to be move dynamically, keep on the move! :-).

Break the problem into pieces... apply DDOS controls on the smaller pieces. :-)

I've tried a most general answer, but there are a lot a of depends, each DDOS mitigation requires a bit of Skin, not tin approach.. Really you need a anti-ddos ninja on your team. ;-)

Take a look at distributed protocols.... DP's maybe the answer for DDOS.

Have fun.

The Unix Janitor
  • 558
  • 1
  • 6
  • 15
0

Let's apply some analysis to your question. A DDoS is like a DoS but with friends. If you want to avoid DDoS explotation you need minimize DoS possibilities. Thanks capitan obvious.

First thing is to do is make a list with what happens in your system and wich resources are affected:

  • A tcp handshake is performed (SYN_COOKIES are affected)
  • A ssl handshake comes later (entropy, cpu)
  • A connection is made to channel layer...

Then monitorize each resource and try to implement a counter-measure:

  • Protect to SYN_FLOOD configuring your kernel params and firewall
  • Use entropy generators
  • Configure your firewall to limit open/closed connection in short time (easy way to minimize ssl handshakes)
  • ...

Separate your big problem (DDoS) in many simple and easy to correct tasks. Hard part is get a detailed list of steps and resources.

Excuse my poor english.

lasizoillo
  • 104
  • 4
  • capitan? captain? – Pang Aug 08 '17 at 04:21
  • Hey @lasizoillo, thanks for the response. As I've mentioned with the other answers, I'm really looking for information on what Daphne supports for this kind of thing. A way to handle large queues, gracefully spawn more workers for a specific channel, or even some sort of throttling. I understand that true DDoS protection is done at a much lower level, but I'm looking for information on Daphne and Django Channels' application level support (think something like DRF's throttling). – Jamie Counsell Aug 08 '17 at 16:50