Associating aiohttp requests with the responses

Question

I would simply like to associate responses from aiohttp asynchronous HTTP requests with an identifier. I am using the following code to hit the API and extract contactproperty object which requires an external field (contacid) in order to call its API:

def get_contact_properties(self, office_name, api_key, ids, chunk_size=100, **params):
    properties_pages = []
    batch = 0
    while True:

        chunk_ids = [ids[i] for i in range(batch * chunk_size + 1, chunk_size * (1 + batch) + 1)]
        urls = ["{}/{}".format(self.__get_base_url(), "contacts/{}/properties?api_key={}".format(contactid, api_key))
                for contactid in chunk_ids]

        responses_raw = self.get_responses(urls, self.get_office_token(office_name), chunk_size)
        try:
            responses_json = [json.loads(response_raw) for response_raw in responses_raw]
        except Exception as e:
            print(e)

        valid_responses = self.__get_valid_contact_properties_responses(responses_json)
        properties_pages.append(valid_responses)


        if len(valid_responses) < chunk_size:  # this is how we know there are no more pages with data
            break
        else:
            batch = batch + 1

ids is a list of ids. The problem is that I do not know which response corresponds to which id so that later I can link it to contact entity using contacid. This is my fetch() function so I was wondering how to edit this function to return the contactid along with output.

async def __fetch(self, url, params, session):
    async with session.get(url, params=params) as response:
        output = await response.read()
        return (output)

async def __bound_fetch(self, sem, url, params, session):
    # Getter function with semaphore.
    async with sem:
        output = await self.__fetch(url, params, session)
        return output

If your question was answered please mark it as [solved](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work) — Ionut Ticus, May 04 '20 at 14:07

Ionut Ticus · Accepted Answer · 2020-05-06T08:59:48.703

You can return the url (or whatever key identifies your request) together with the output.

Regarding using the data, I think you should read the response directly as JSON, especially since aiohttp can do this for you automatically.

async def __fetch(self, url, params, session):
    async with session.get(url, params=params) as response:
        try:
            data = await response.json()
        except ValueError as exc:
            print(exc)
            return None
        return data

async def __bound_fetch(self, sem, url, params, session):
    # Getter function with semaphore.
    async with sem:
        output = await self.__fetch(url, params, session)
        return {"url": url, "data": data}

You did not post the get_responses function but I'm guessing something like this should work:

responses = self.get_responses(urls, self.get_office_token(office_name), chunk_size)

Responses will be a list of {"url": url, data: "data"} (data can be None for invalid responses); however with the code above one invalid request will not affect the others.

Do you mean url instead of params? because params is empty but url has the contact id. Can you please complete your answer? because __bound_fetch returns a dict so how can I pass it to json.loads??? responses_json = [json.loads(response_raw) for response_raw in responses_raw] — jack mh, May 05 '20 at 11:12

Associating aiohttp requests with the responses

1 Answers1