Problem
I have purchase service that users can use to buy/rent digital assets like game, media, movies... When purchase event happened, I create a job and schedule it to run at calculated expired date to remove key for such asset.
Everything works. But it would be better if I can group those jobs that will run in the same agenda db scan into 1 batch job to remove multiple keys.
This will reduce significant amount of db read/write/delete operations in both keys and agenda collection, it also increases the free memory at most of the time as instead of storing 100+ jobs to run in a scan, it stores just 1 job to remove 100+ keys.
Research
The closest feature I found in Agenda repo is unique(). Which allows user to find and modify the existing job that matches the fields defined in unique(). If it can concat new jobs to the existing job, that will solve my case.
Implementation
Before diving in and modify the package I want to check if there are already people solved the problem I have and have some thoughts to share.
Another solution without touching the package is to create an in-memory dictionary to accumulate jobs for a specific db scan with this strategy:
dict = {}
//if key expires in 1597202228 then put to dict slot:
dict = {
1597300000: [jobA]
}
//another key expires in 1597202238 then put to the same slot:
dict = {
1597300000: [jobA,jobB]
}
//the latch condition to put job batch into agenda:
if dict_size == dict_allocated_memory then put the whole dict into db.
if a batch_size = batch_limit then put the batch into db and remove the batch in dict.
if the batch is going to expire in the next db scan then put the batch (it may be empty, has a few jobs...) into db and remove the batch in dict.
