0

I wonder if this has ever come up before?

I have an app that downloads tons of information from our GitHub Enterprise instance (not currently using github3.py but I am thinking of converting it over to use github3.py). Basically, it crawls 189 repos and then for each repo, it pulls branches, tags, and commits. This is obviously pretty slow, because it has to do so many HTTP requests serially.

I wonder if github3.py could be made to return grequest objects so that I could have grequests do the requests in parallel? Or perhaps an alternative would be to wrap and hide grequests by having the ability to set a "concurrency level" in github3.py and have it take care of doing requests in parallel. Or maybe a context manager:

with github3.parallel():
    tags = pull_tags(git_repo)
    branches = pull_branches(git_repo)

I recognize that it could be quite challenging to come up with an API that works well with parallelization since it's such a different paradigm. Which is why I didn't want to clutter up the issue tracker with this.

Marc Abramowitz
  • 3,447
  • 3
  • 24
  • 30

1 Answers1

1

This would be extraordinarily difficult. grequests and other wrappers around requests do not provide a Session API (because requests' Sessions aren't as threadsafe as they should be -- i.e., they're not). If they offered a way of doing this, that might work. The best way to do this, is to create multiple GitHub instances, one in each worker greenlet/thread, and use them in the context of each thread. By doing so, you'll have a Session per greenlet and your operations should be safe (so long as you don't pass your objects between threads).

Ian Stapleton Cordasco
  • 26,944
  • 4
  • 67
  • 72