Play 2.2.3 on Heroku with clearDB - has exceeded the 'max_user_connections' resource

Question

My Play app was working fine for a while on Heroku with clearDB. Then it crashed with the error

has exceeded the 'max_user_connections' resource

I followed the instructions from this post

playframework 2.0 - exceeded max_user_connections on database evolutions?

However, the problem was not resolved. I even tried adding this configuration to the Procfile.

I'm able to connect to my database through MySQL workbench, so the problem doesn't seem to persist when my application is down.

My log output while trying to restart is as follows.

State changed from crashed to starting
Starting process with command `target/universal/stage/bin/listbucket -Dhttp.port=${PORT} -Ddb.default.driver=com.mysql.jdbc.Driver -Ddb.default.url=${CLEARDB_DATABASE_URL} -Ddb.default.partitionCount=1 -Ddb.default.maxConnectionsPerPartition=5 -Ddb.default.minConnectionsPerPartition=5`
Picked up JAVA_TOOL_OPTIONS:  -Djava.rmi.server.useCodebaseOnly=true
Play server process ID is 2
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 10. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 9. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 8. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 7. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 6. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 5. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 4. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 3. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 2. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 1. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
    at play.api.Configuration$.play$api$Configuration$$configError(Configuration.scala:92)
    at play.api.Configuration.reportError(Configuration.scala:570)
Configuration error: Configuration error[Cannot connect to database [default]]
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.immutable.List.foreach(List.scala:318)
Oops, cannot start the server.
    at scala.collection.immutable.List.foreach(List.scala:318)
    at play.api.Play$$anonfun$start$1.apply(Play.scala:88)
    at play.utils.Threads$.withContextClassLoader(Threads.scala:18)
    at play.core.StaticApplication.<init>(ApplicationProvider.scala:52)
    at play.api.Play$$anonfun$start$1$$anonfun$apply$mcV$sp$1.apply(Play.scala:88)
    at play.api.db.BoneCPPlugin$$anonfun$onStart$1.apply(DB.scala:243)
    at play.api.Play$$anonfun$start$1$$anonfun$apply$mcV$sp$1.apply(Play.scala:88)
    at play.core.server.NettyServer$$anonfun$main$3.apply(NettyServer.scala:274)
    at play.core.server.NettyServer$$anonfun$main$3.apply(NettyServer.scala:279)
    at play.core.server.NettyServer$.main(NettyServer.scala:274)
    at play.core.server.NettyServer$.createServer(NettyServer.scala:243)
    at play.core.server.NettyServer.main(NettyServer.scala)
    at play.api.Play$$anonfun$start$1.apply$mcV$sp(Play.scala:88)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.Option.map(Option.scala:145)
    at play.api.Play$$anonfun$start$1.apply(Play.scala:88)
    at play.api.db.BoneCPPlugin$$anonfun$onStart$1.apply(DB.scala:252)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at play.api.db.BoneCPPlugin.onStart(DB.scala:243)
    at play.api.Play$.start(Play.scala:87)
State changed from starting to crashed
Process exited with status 255
State changed from crashed to starting
Starting process with command `target/universal/stage/bin/listbucket -Dhttp.port=${PORT} -Ddb.default.driver=com.mysql.jdbc.Driver -Ddb.default.url=${CLEARDB_DATABASE_URL} -Ddb.default.partitionCount=1 -Ddb.default.maxConnectionsPerPartition=5 -Ddb.default.minConnectionsPerPartition=5`
Picked up JAVA_TOOL_OPTIONS:  -Djava.rmi.server.useCodebaseOnly=true
Play server process ID is 2
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and t
rying again. Attempts left: 10. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 9. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 8. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 7. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 6. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 5. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 4. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 3. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 2. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)
[error] c.j.b.h.AbstractConnectionHook - Failed to acquire connection to jdbc:mysql://us-cdbr-iron-east-01.cleardb.net/heroku_f176dd782ce596b?reconnect=true Sleeping for 1000ms and trying again. Attempts left: 1. Exception: null.Message:User 'b6cf16f10a7a77' has exceeded the 'max_user_connections' resource (current value: 10)

Heroku has pretty much been one big headache after another, so any help would be greatly appreciated, as this is pretty much the last straw for me with Heroku.

Answer 1

You need to edit the partition connections for db.default in your *.conf file.

See documentation here.

You will be interested in these three.

# In order to reduce lock contention and thus improve performance, 
# each incoming connection request picks off a connection from a 
# pool that has thread-affinity. 
# The higher this number, the better your performance will be for the 
# case when you have plenty of short-lived threads. 
# Beyond a certain threshold, maintenance of these pools will start 
# to have a negative effect on performance (and only for the case 
# when connections on a partition start running out).
db.default.partitionCount=2

# The number of connections to create per partition. Setting this to 
# 5 with 3 partitions means you will have 15 unique connections to the 
# database. Note that BoneCP will not create all these connections in 
# one go but rather start off with minConnectionsPerPartition and 
# gradually increase connections as required.
db.default.maxConnectionsPerPartition=5

# The number of initial connections, per partition.
db.default.minConnectionsPerPartition=5