Python Requests - Batch API calls

Question

I am working on an API to push records from a database to endpoint.

I have managed to write a piece of code which grabs the records, it's parsing them and finally pushing to API and works just fine.

It manages to push around 400 req/min and I was wondering if I can batch this requests to make it a bit more performant, but I can't wrap my head around how this can be achieved.

The API call url is:

http://call-url.com/service/{id}

Consider payload for id:

id = 101

{
"stars":3
"position":5
"date":"2002-04-30T10:00:00+00:00"
}
url = http://call-url.com/service/101

I am using python and requests module to push the records.

As we speak I am grabbing the records and parsing load for each individual id and pushing them.

I bumped into asyncio and threading so far, but I need to ensure I don't push the same request twice.

Is it possible to push more than 1 record at once?

Thank you,

Iurii Tkachenko · Answer 1 · 2019-07-10T20:06:25.407

2

You can utilize any AMQP message broker. RabbitMQ for example. The concept is simple, just check the tutorial. Split your script to main.py (read DB, prepares payloads and push them to the queue) and worker.py (get payload from the queue and send to API), and then just spawn as many worker.py processes as you need...

edited Jul 10 '19 at 20:06

answered Jul 10 '19 at 19:22

Iurii Tkachenko

3,106
29
34

1

Appreciate it, yes it makes sense and it looks like how I was visualizing it. I will give it a go. – madlicksxxx Jul 10 '19 at 20:04
@madlicksxxx make sure you don't miss Message acknowledgment section. It's very important. – Iurii Tkachenko Jul 10 '19 at 20:08

Python Requests - Batch API calls

1 Answers1