1

I did a quick google and was not able to find any definitive statements. Basically, I am trying to read events from SQS and process them, and the asio is expected to be handled by greenlets.

So what is the granularity of boto3 can be used in a greenlet? Do I need session/client per greenlet? Do I need additional mutex?

xis
  • 24,330
  • 9
  • 43
  • 59

2 Answers2

2

Boto3 no but you could monkey patch botocore.

In the top of your modules you would always need to monkey patch first and afterwards use botocore session, example:

from gevent import monkey
monkey.patch_all()

import botocore.session


def fetch_file(bucket, key):
    session = botocore.session.get_session()
    client = session.create_client('S3')
    return client.get_object(
       Bucket=bucket, 
       Key=key
      )

  result = gevent.spawn(fetch_file('mybucket', 'sd.txt'))

UPDATE 03.04.2021

I just checked and verified that gevent monkey_patch method works with boto3:

from gevent import monkey, spawn, joinall
monkey.patch_all()

import boto3 # make sure you import after doing the monkey patch


def ec2_region(region_name):
    instances = 0
    result = 'succeeded'
    ec2 = boto3.Session().resource('ec2', region_name=region_name)

    try:
        result_instances = sum(1 for _ in ec2.instances.all())
    except:
        result = 'failed'

    return f'{region_name:20}\t{result:10}\t{instances}'


regions = boto3.session.Session().get_available_regions('ec2')
jobs = [spawn(ec2_region, region_name) for region_name in regions]
joinall(jobs)

print(*[job.value for job in jobs], sep='\n')

It takes around 2-3 seconds.

Without gevent, e.g.:

import boto3


def ec2_region(region_name):
    instances = 0
    result = 'succeeded'
    ec2 = boto3.Session().resource('ec2', region_name=region_name)

    try:
        result_instances = sum(1 for _ in ec2.instances.all())
    except:
        result = 'failed'

    return f'{region_name:20}\t{result:10}\t{instances}'


regions = boto3.session.Session().get_available_regions('ec2')
jobs = [ec2_region(region_name) for region_name in regions]

print(*jobs, sep='\n')

takes around 17seconds.

Bobby Donchev
  • 335
  • 1
  • 5
  • 2
    Shouldn't monkey patching handle any network io underneath automatically? Also doesn't boto3 use botocore underneath? – jetru Apr 11 '18 at 07:50
  • Yes boto3 uses botocore but like I said you cant patch boto3. You can ditch boto3 and use botocore yourself and patch in order to use greenlets – Bobby Donchev Jan 21 '19 at 13:41
  • gevent should monkey_path boto3 directly – Narcisse Doudieu Siewe May 30 '20 at 12:03
  • Any explanation for this answer? why woudlnt boto3 work if the patch is in the socket / requests level? – yogi Mar 17 '21 at 12:54
  • 1
    I just updated my answer after verifying again. When I first tried monkey patching boto3 in 2017 I wasnt able to. Not sure if boto3 changed or gevent changed but now it seems to work – Bobby Donchev Apr 03 '21 at 14:27
0

gevent monkey patch does patch all of the socket calls underneath automatically, as long as botocore uses those socket calls. And boto3 does use botocore.

microe14
  • 2,711
  • 1
  • 15
  • 5