-1

I am trying to make a simple general purpose multi-threaded async downloader in python.How many parallel connections can be generally be made to a server with minimum risk of being banned or rate limited.

I am aware that network will be a limiting in some cases but lets assume in this case that network isn't an issue in this case for the sake of discussion.I/O is also done asynchronously.

According to Browserscope , browsers make a maximum of 17 connections at a time.

However according to my research , most download managers download files in multi-part and make 8+ connections per file.

1.How many files can be downloaded at a time ?

2.How many chunks for a single can be downloaded at one time ?

3.What should be the minimum size of those chunks to make it worth creating the overhead of creating parallel connections ?

user2650277
  • 6,289
  • 17
  • 63
  • 132
  • Why the down vote?? – user2650277 Feb 20 '18 at 15:57
  • This depends on the server. It's impossible to tell without either its config or brute testing. – Zac67 Feb 20 '18 at 18:13
  • @Zac67 read the post again `general purpose multi-threaded async downloader` – user2650277 Feb 20 '18 at 18:29
  • You're asking about an *unknown* server and quote a statistic applying to *browsers*. What are you aiming at? – Zac67 Feb 20 '18 at 18:34
  • @Zac67 have you heard about download managers or download accelerators.The software will download files faster by downloading chunks in parallel.Browser stats is a starting point(at least we know 17 is undisguisable from a real browser.I am looking for the upper bound – user2650277 Feb 20 '18 at 18:56
  • I've been using download managers for more than 20 years. There's simply no generally safe number you're asking for. btw: A good user tracking can always identify download managers, no matter what you try. – Zac67 Feb 20 '18 at 19:04

1 Answers1

0

It depends.

While some servers tolerate a high number of connections, others don't. General web servers might be more on the high side (low two digit), file hosters might be more sensitive.

There's little to say unless you can check the server's configuration or just try and remember for the next time when your ban has timed out.

You should however watch your bandwidth. Once you max out your access line there's no gain in further increasing the connections.

Zac67
  • 2,761
  • 1
  • 10
  • 21