How can I create a disk-based delayed queue in Go?
I am writing a Go program to take certain actions on data after certain time intervals. Here is a stripped down version.
func IncomingJob (data MyStruct) {
//Run immediately
dosomething(&data, 1)
time.Sleep(5 * time.Minute)
//Run after 5 minutes has passed from initial arrival
dosomething(&data, 2)
time.Sleep(5 * time.Minute)
//Run after 10 minutes has passed from initial arrival
dosomething(&data, 3)
time.Sleep(50 * time.Minute)
//Run after 60 minutes has passed from initial arrival
dosomething(&data, 4)
}
This func would be initialized as a goroutinue.
go IncomingJob(data)
The problem I am having is that each instance of IncomingJob runs for 60 minutes and the MyStruct object stays in memory for that duration. The total memory usage is pretty huge. If I want to support 1 million runs per hour, that means at any given time, there are 1 million MyStruct objects in memory waiting, doing nothing. I can use time.AfterFunc for this as well, but that doesn't change anything with respect to memory usage.
Is there some disk backed FIFO queue/buffer implementation in Go? That way I could have 3 queues (for different intervals) and poll it and sleep until its the correct time for the polled data to be reprocessed. This way I would lose some CPU cycles in serialization, add I/O latency to an otherwise in-memory application but save a massive amount of memory.
Update: time.AfterFunc uses a lot less memory.
func IncomingJob (data MyStruct) {
//Run immediately
dosomething(&data, 1)
time.AfterFunc(5 * time.Minute, func() {
//Run after 5 minutes has passed from initial arrival
dosomething(&data, 2)
time.AfterFunc(5 * time.Minute, func() {
//Run after 10 minutes has passed from initial arrival
dosomething(&data, 3)
})
time.AfterFunc(50 * time.Minute, func() {
//Run after 50 minutes has passed from initial arrival
dosomething(&data, 4)
})
})
}
