0

I'm currently trying to make my requests faster by multithreading them but I'm not sure how to do it the way I want. I know about grequests but they seem to require a URL list. I have code with a starting number contained in URL and would like all threads to stop after getting a status_code of 200

I have tried to accomplish this by grequests but couldn't make it work. Also tried threading but don't know how to stop all threads after working URL was found

import requests
import webbrowser

def url_request(number):
    url = "http://website.com/download/" + str(number) + ".zip"
    r = requests.head(url)
    if r.status_code == 404:
        print(url + " - 404 Not Found!")
        number += 1
        url_request(number)
    elif r.status_code == 200:
        webbrowser.open(url)
        print(url + " - 200 Found!")

if __name__ == "__main__":
    url_request(int(input("Starting number: ")))

What I want the code to do is execute multiple request.head at once with a number after "Starting number" and will stop after one of the threads finds url with status_code 200.

kreny
  • 18
  • 4
  • 1
    Why don't you pre-build the list of urls and then send all the requests in parallel? – rdas Apr 08 '19 at 16:46
  • @DroidX86 Well, I could try that, of course. But I wanted to be able to set the starting point and not have any preexisting lists. – kreny Apr 08 '19 at 17:04
  • If you don't want a pre-existing lists, try writing a generator that will generate the URLs! – rdas Apr 08 '19 at 17:05

1 Answers1

0

Ok, figured it out. Thanks for your advice.

Here's the code:

from gevent import monkey
monkey.patch_all()
import grequests
import webbrowser


def url_request_threaded(startnumber, stopnumber):
    urls = []
    for i in range(startnumber, stopnumber):
        urls.append("http://website.com/download/" + str(i) + ".zip")
    gr = (grequests.head(url, stream=False) for url in urls)
    gresponses = grequests.imap(gr, size=10)
    try:
        for response in gresponses:
            if response.status_code == 404:
                print(response.url + " - 404 Not Found!")
            elif response.status_code == 200:
                webbrowser.open(response.url)
                print(response.url + " - 200 Found!")
                raise SystemExit
    except SystemExit:
        pass


if __name__ == "__main__":
    while True:
        try:
            startn = input("Starting number: ")
            startn = int(startn)
            stopn = input("End number: ")
            stopn = int(stopn)
            url_request_threaded(b, v, startn, stopn)
        except ValueError:
            print("Must be a number!")
            continue
        break
kreny
  • 18
  • 4