0

I have some nightly jobs that are running on EC2 and the number of machines is scaled by the number of messages in SQS. My process requires reads from a Postgres RDS database. Now these are the issues I am facing.

  1. Not able to scale beyond a certain number because of the unavailability of connections.
  2. I tried creating a connection pool using pgbouncer, and tried with different settings as well, but it's missing a lot of data on the resultant set.
John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
najeeb
  • 813
  • 12
  • 25

1 Answers1

1

Make your postgresql RDS install multi AZ. Then you can make read replicas on demand and scale read performance with your load.

To answer the comments:

  • Some extra "plumbing" is required to make the connections to the read replica. Maybe route53 dynamically updated records as the scaling happens or something like haproxy
  • The reason I mention multi AZ is that this would help prevent downtime during an auto scaling event bringing up the read replica

It would be simpler (but more costly) to permanently bring up a read replica and use DNS round robin to share the load

See https://aws.amazon.com/blogs/aws/amazon-rds-announcing-read-replicas/ for information on read replicas

Vorsprung
  • 32,923
  • 5
  • 39
  • 63
  • 1
    Using read replicas is a good strategy for read-heavy workloads, but the application would need to know how to send read requests to a replica and write requests to the master server. Multi-AZ is not required for Read Replicas. See: [PostgreSQL Read Replicas documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.PostgreSQL) – John Rotenstein Apr 01 '16 at 10:52
  • Hi Vorsprung, I attempted that, but with my Django application I could only able to point to either one (master/slave). I didn't see them sharing the load. – najeeb Apr 01 '16 at 12:00