How to send an etag or last modified while making a request with aiohttp session.get

Question

I am loading multiple feed urls every minute
I want to send a http get request, get 200 status code and full data if the data has changed since the last time I loaded it
I want the http status code 304 and no response body if the data has not changed since the last time I loaded it
Python feedparser offers HERE this functionality if I use their library to send GET requests
How do I do this using aiohttp library

How do I send etag and last modified in the GET request

async with session.get(url) as response:
    text = await response.text()
    print(response.headers.get('etag'), response.headers.get('Last-Modified'))

How do I send etag and last modified and simulate behavior similar to the library above?

UPDATE 1

Here is some detailed code

import asyncio
import aiohttp

async def load_feed(session, url):
    # Keep this an empty string for the first request
    etag = 'fd31d1100c6390bd8a1f16d2703d56c0'
    # Keep this an empty string for the first request
    last_modified='Mon, 11 May 2020 22:27:44 GMT'
    try:
        async with session.get(url, headers={'etag': etag, 'Last-Modified': last_modified}) as response:
            t = await response.text()
            print(response.headers.get('etag'), response.headers.get('Last-Modified'), response.status, len(t), response.headers)
    except Exception as e:
        print(e)

async def load_feeds():
    try:
        async with aiohttp.ClientSession() as session:
            tasks = []
            for url in ['https://news.bitcoin.com/feed/']:
                task = asyncio.ensure_future(load_feed(session, url))
                tasks.append(task)
            await asyncio.gather(*tasks, return_exceptions=True)
    except:
        pass

asyncio.get_event_loop().run_until_complete(load_feeds())

Expectation:

Send a request the first time without headers
Get a response code 200 with etag and Last-modified and full response
Send a request again with etag and Last-modified
Get a response code 304 with no response body

What is happening - I am getting status code 200 and full response body every time

Tried a few other urls too like https://coindesk.com/feed

Could you add a bit more detail? I'm not sure what you're trying to do that doesn't yet work — PirateNinjas, May 12 '20 at 08:22
i want to send a request with aiohttp with etag and last-modified, it should send me a 304 response code with no actual response in case the data has not changed, otherwise it sends 200 response code wilth data — PirateApp, May 12 '20 at 08:29
I don't think aiohttp supports this by itself; however you can store the headers (`ETag` and `Last-Modified`) when first making the request (which will return 200) and reuse them for subsequent requests (which should return 304). Of course you need to update the stored headers when you receive a new 200 response. — Ionut Ticus, May 12 '20 at 09:34

score 4 · Accepted Answer · answered May 12 '20 at 14:12

Last-Modified is a response header, for a request you would use If-Modified-Since:

async def load_feed(session, url):
    headers = {
        "ETag": "fd31d1100c6390bd8a1f16d2703d56c0",
        "If-Modified-Since": "Mon, 11 May 2020 22:27:44 GMT"
    }
    try:
        async with session.get(url, headers=headers) as response:
            t = await response.text()
            print(response.headers.get("ETag"), response.headers.get('Last-Modified'), response.status, len(t))
    except Exception as e:
        print(e)

Output (notice status=304):

"fd31d1100c6390bd8a1f16d2703d56c0" Mon, 11 May 2020 22:27:44 GMT 304 0

How to send an etag or last modified while making a request with aiohttp session.get

1 Answers1