How to batch process from Postgres read replica with multiple java instances?

Asked Dec 19 '21 at 00:40

Active Dec 19 '21 at 00:40

Viewed 105 times

I am currently trying to write a scheduler in Java which will read from an AWS Postgres read replica in batches and do some processing. This scheduler will be run on a cluster of Java instances. The current issue I am having is that I am unable to tell whether a row has been processed yet or not.

I am thinking about using a “SELECT * from table Limit 1 FOR UPDATE SKIP LOCKED” but I am wondering if:

Is it possible to SELECT FOR UPDATE on a read replica
How would I release this lock?
If I cannot use this, is there any other way I can coordinate the batch processing on a read replica with multiple Java instances. (I.e would spring batch be able to this?)

asked Dec 19 '21 at 00:40

Alan Shiau

Are you updating the row after it has been read by your java scheduler? – Shreyash Solanke Dec 19 '21 at 06:32
No I am not. I’m just reading from the row – Alan Shiau Dec 19 '21 at 10:44
With Spring Batch, you can create different job instances and make each job instance process a distinct data set. Another option is to use a partitioned step where each worker is assigned a different data set. – Mahmoud Ben Hassine Dec 21 '21 at 21:07

How to batch process from Postgres read replica with multiple java instances?

0 Answers0