13

I am getting FATAL: the database system is shutting down errors while creating PostgreSQL JDBC connections to a PostgreSQL 9.2 server. The specific exception path I'm getting from JDBC is here:

Caused by: org.postgresql.util.PSQLException: FATAL: the database system is shutting down
    at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173)
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64)
    at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136)
    at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
    at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
    at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31)
    at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
    at org.postgresql.Driver.makeConnection(Driver.java:393)
    at org.postgresql.Driver.connect(Driver.java:267)

From various log files (from PostgreSQL, from our management layer, from the app using PostgreSQL), I do not see any database shutdown actually happening (other connections are created to PostgreSQL as usual, no shutdown was initiated from our management layer, etc.), however in the PostgreSQL server logs I do see the error message with the same timestamp:

2014-06-16 12:30:00.736 GMT LOG:  connection received: host=127.0.0.1 port=38530
2014-06-16 12:30:00.737 GMT FATAL:  the database system is shutting down

Researching online, I learned that this error message is used whenever PostgreSQL shuts down connections.

Why would PostgreSQL refuse to give me a new connection? Could this be caused by some sort of resource contention? How can I get more information about the error from PostgreSQL itself?

Dan
  • 7,155
  • 2
  • 29
  • 54

4 Answers4

12

This issue turned out to be caused by a very bad misuse of PostgreSQL where our server had booted two PostgreSQL instances on the same data directory (it had deleted postmaster.pid and used a new port number, so the normal safeguards against this behavior were bypassed) which explains why the logs did not hold any useful information -- they were being overwritten by the PostgreSQL instance which wasn't refusing connections. The actual issue was caused by a complex interaction between the competing PostgreSQL instances, and I hope nobody else runs into this issue as well!

Dan
  • 7,155
  • 2
  • 29
  • 54
  • 8
    What was the solution that fixed it? – Phil Mar 03 '20 at 18:29
  • can you please, provide the solution here? – Juhi Dhameliya Dec 18 '20 at 08:33
  • The solution in my case was not running two PostgreSQL servers on the same set of files. If you don't manually delete the `postmaster.pid` file, or are more careful than we were being when you do it, then PostgreSQL is designed to prevent you from doing this. – Dan Dec 19 '20 at 17:39
1

We encountered this problem when the server was restarted. In this case the reboot did not initiate the connection with PostgreSql. Just check the status: "/etc/init.d/postgresql status" if the return is: "no server running" just start: "/etc/init.d/postgresql start"

Thyago César Souto
  • 11
  • 1
1

Maybe there is not enough space on hard drive.

Amir.S
  • 719
  • 8
  • 15
1

SOLUTION:

  1. brew remove postgresql
  2. Temporary rename the lock files: mv /tmp/.s.PGSQL.5432.lock /tmp/BK.s.PGSQL.5432.lockand mv /tmp/.s.PGSQL.5432 /tmp/BK.s.PGSQL.5432
  3. brew install postgresql

Enjoy!

cazuba
  • 329
  • 2
  • 5