How to share an aiohttp.ClientSession() between processes?

Question

I'm trying to create a shared resource in which I can use an aiohttp.ClientSession() object to fetch certain data. This allows me to use parallelization to speed up the actual calculations.

However, when I try to create a manager in which I use aiohttp.ClientSession, an error is raised: TypeError: no default __reduce__ due to non-trivial __cinit__.

I'm not sure what is going on and perhaps someone would be able to help me out on what this error means and how I could actually initialize a manager that used an aiohttp.ClientSession.

import asyncio
import aiohttp

# import nest_asyncio    

from multiprocessing.managers import BaseManager

class Session:
    def __init__(self, loop):
        asyncio.set_event_loop(loop)
        self.session = aiohttp.ClientSession()

class MyManager(BaseManager):
    pass

# nest_asyncio.apply()

MyManager.register('Session', Session)
loop = asyncio.get_event_loop()

with MyManager() as manager:
    session = manager.Session(loop)

As a fallback solution, I could use requests, which seems to have no problems when registered in a BaseManager. However, this would seriously bottleneck my process, as I have a fairly large workload running over 60+ parallel processes in which each data call takes a fairly long time to complete.

import requests

from multiprocessing.managers import BaseManager

class Session:
    def __init__(self):
        self.session = requests.Session()

class MyManager(BaseManager):
    pass

MyManager.register('Session', Session)

with MyManager() as manager:
    session = manager.Session()

I hope someone can help me out and many thanks in advance!

Try to use raw `aiohttp.ClientSession()` instead of including `asyncio.set_event_loop(loop)`. Basically you shouldn't be able to share that event loop. Or at least it makes no sense. If you cannot even share the `aiohttp.ClientSession()`, it means it is totally unpickable. — Sraw, Jul 02 '20 at 17:36
@Sraw many thanks! I thought it would be unpickable, but with your suggestion my manager seems to be more than okay with using aiohttp. — Nesterion, Jul 02 '20 at 19:10

score 0 · Answer 1 · answered Jul 02 '20 at 19:10

Thanks Sraw! Based on your suggestion I managed to create a manager without errors! According to the asyncio documentation, a convent and preferred way to create a loop is by simply calling asyncio.run().

import asyncio
import aiohttp

from multiprocessing.managers import BaseManager

class Session:
    def __init__(self, beta_engine=False):        
        process = self._init_session()
        asyncio.run(process)
        
    async def _init_session(self):
        self.session = aiohttp.ClientSession()
        
class MyManager(BaseManager):
    pass

MyManager.register('Session', Session)

with MyManager() as manager:
    session = manager.Session()

How to share an aiohttp.ClientSession() between processes?

1 Answers1