Getting Internal Server Error on prisma deploy

Question

I have a Postgres database on Heroku, upon deploying the data model by doing prisma deploy often times the following error is produced.

ERROR: Whoops. Looks like an internal server error. Search your server logs for request ID: local:cjxrmcnpx00hq0692zuwttqwv

{
 "data": {
   "addProject": null
 },
 "errors": [
   {
     "message": "Whoops. Looks like an internal server error. Search your server logs for request ID: local:cjxrmcnpx00hq0692zuwttqwv",
     "path": [
       "addProject"
     ],
     "locations": [
       {
         "line": 2,
         "column": 9
       }
     ],
     "requestId": "local:cjxrmcnpx00hq0692zuwttqwv"
   }
 ],
 "status": 200
}

and on checking the Docker logs I am seeing this erorr:

Jul 14, 2019 12:18:34 PM org.postgresql.Driver connect
prisma_1  | SEVERE: Connection error: 
prisma_1  | org.postgresql.util.PSQLException: FATAL: too many connections for role "bcueventxumaik"  
prisma_1  |     at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
prisma_1  |     at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2566)
prisma_1  |     at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:131)        
prisma_1  |     at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:210)
prisma_1  |     at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
prisma_1  |     at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
prisma_1  |     at org.postgresql.Driver.makeConnection(Driver.java:452)
prisma_1  |     at org.postgresql.Driver.connect(Driver.java:254)
prisma_1  |     at slick.jdbc.DriverDataSource.getConnection(DriverDataSource.scala:101)
prisma_1  |     at slick.jdbc.DataSourceJdbcDataSource.createConnection(JdbcDataSource.scala:68)      
prisma_1  |     at slick.jdbc.JdbcBackend$BaseSession.<init>(JdbcBackend.scala:453)
prisma_1  |     at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:46)
prisma_1  |     at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:37)
prisma_1  |     at slick.basic.BasicBackend$DatabaseDef.acquireSession(BasicBackend.scala:249)        
prisma_1  |     at slick.basic.BasicBackend$DatabaseDef.acquireSession$(BasicBackend.scala:248)       
prisma_1  |     at slick.jdbc.JdbcBackend$DatabaseDef.acquireSession(JdbcBackend.scala:37)
prisma_1  |     at slick.basic.BasicBackend$DatabaseDef$$anon$2.run(BasicBackend.scala:274)
prisma_1  |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)    
prisma_1  |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)    
prisma_1  |     at java.lang.Thread.run(Thread.java:748)
prisma_1  |
prisma_1  | Exception in thread "main" org.postgresql.util.PSQLException: FATAL: too many connections 
prisma_1  |     at org.postgresql.core.v3.QueryExecutorImpl.readStartupMessages(QueryExecutorImpl.java:2566)prisma_1  |     at org.postgresql.core.v3.QueryExecutorImpl.<init>(QueryExecutorImpl.java:131)prisma_1  |     at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:210)
prisma_1  |     at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)prisma_1  |     at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
prisma_1  |     at org.postgresql.Driver.makeConnection(Driver.java:452)
prisma_1  |     at org.postgresql.Driver.connect(Driver.java:254)prisma_1  |     at slick.jdbc.DriverDataSource.getConnection(DriverDataSource.scala:101)
prisma_1  |     at slick.jdbc.DataSourceJdbcDataSource.createConnection(JdbcDataSource.scala:68)
prisma_1  |     at slick.jdbc.JdbcBackend$BaseSession.<init>(JdbcBackend.scala:453)
prisma_1  |     at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:46)
prisma_1  |     at slick.jdbc.JdbcBackend$DatabaseDef.createSession(JdbcBackend.scala:37)
prisma_1  |     at slick.basic.BasicBackend$DatabaseDef.acquireSession(BasicBackend.scala:249)
prisma_1  |     at slick.basic.BasicBackend$DatabaseDef.acquireSession$(BasicBackend.scala:248)
prisma_1  |     at slick.jdbc.JdbcBackend$DatabaseDef.acquireSession(JdbcBackend.scala:37)
prisma_1  |     at slick.basic.BasicBackend$DatabaseDef$$anon$2.run(BasicBackend.scala:274)
prisma_1  |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
prisma_1  |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
prisma_1  |     at java.lang.Thread.run(Thread.java:748)
prisma_prisma_1 exited with code 1

The error is saying too many connections but I am firing prisma deploy from only one terminal and at the same time I am able to connect to the database using PgAdmin4. Moreover, the database seems to be perfectly reachable as I am able to ping the database from inside the container.

PS. Updated the docker logs as earlier on running docker logs -f processid I was getting older logs but now upon building the container again using docker-compose up I got the lastest logs

can you ping `ec2-54-228-246-214.eu-west-1.compute.amazonaws.com`? — arnonuem, Jul 10 '19 at 11:43
I am able to ping from inside the Docker container. I would replicate the same when next time prisma deploy won't work. — Neeraj Sewani, Jul 10 '19 at 12:16
@arnonuem Yes I am able to ping `ec2-54-228-246-214.eu-west-1.compute.amazonaws.com` from inside the docker container when `prisma deploy` stucks and later throws _internal server error_ — Neeraj Sewani, Jul 14 '19 at 11:18
But at the same time, I am able to connect from PgAdmin and from the terminal I am making one connection at a time. — Neeraj Sewani, Jul 15 '19 at 09:25

score 0 · Answer 1 · answered Jul 16 '19 at 08:44

As the error clearly states there are too many connections to the database. So we need to investigate how many connections there are, who is creating them and why they are created. In order to either limit the consumers or increase the amount of available connections.

First we can start by using the heroku CLI to check the amount of used and available connections:

$ heroku pg:info

=== DATABASE_URL
Plan:                  Private 2
Status:                Available
HA Status:             Available
Data Size:             2.23 GB
Tables:                83
PG Version:            10.1
Connections:           26/400
Connection Pooling:    Available

For more information on how to investigate heroku postgres databases see: https://devcenter.heroku.com/articles/heroku-postgresql#pg-info

To further investigate who is connected to your database you can either use psql or pgAdmin. If using pgAdmin you can select the database, click on the dashboard tab and select the server activity panel on the bottom of the page revealing all connected sessions. If using psql you could write a select like this:

SELECT pid as process_id, 
       usename as username, 
       datname as database_name, 
       client_addr as client_address, 
       application_name,
       backend_start,
       state,
  FROM pg_stat_activity;

For a more detailed how to see: https://dataedo.com/kb/query/postgresql/list-database-sessions

By now you probably identified who is creating the connections to your database and can limit the client to use less (or increase the amount of available database connections).

One possible consumer of database connections is the prisma server itself of course. The prisma config luckily provides a setting to limit database connections.

The connectionLimit property in PRISMA_CONFIG determines the number of database connections a Prisma service is going to use.

You can read more about it here: https://www.prisma.io/docs/prisma-server/database-connector-POSTGRES-jgfr/#managing-database-connections

If you are using heroku to run the docker container with your prisma server a PRISMA_CONFIG could look like this:

port: $PORT
managementApiSecret: ${PRISMA_MANAGEMENT_API_SECRET}
databases:
  default:
    connector: postgres
    migrations: true
    connectionLimit: 2
    uri: ${DATABASE_URL}?ssl=1

I hope this structured approach helped. Let me know if you need more clarification. If so please provide details regarding the nature of the existing database connections.

Sorry for getting back after almost a fortnight but in Heroku, I never reach the max connections limit and I have checked the connections from PgAdmin and only PgAdmin and Prisma are making connections to the database. Moreover, I have noticed in the last few months that when a database is used for a certain amount of time then Prisma throws this error as I have till now destroyed 3 databases just because of Prisma. — Neeraj Sewani, Jul 28 '19 at 12:28

score 0 · Answer 2 · answered Jun 03 '20 at 13:13

0

run this command

docker logs <YOUR_PRISMA_CONTAINER_NAME>

answered Jun 03 '20 at 13:13

TheEhsanSarshar

2,677
22
41

score 0 · Answer 3 · answered Sep 12 '22 at 19:54

use pooling:

import dotenv from 'dotenv' 
dotenv.config()
import { PrismaClient } from '@prisma/client'

// add prisma to the NodeJS global type
interface CustomNodeJsGlobal extends NodeJS.Global {
  prisma: PrismaClient
}

// Prevent multiple instances of Prisma Client in development
declare const global: CustomNodeJsGlobal

const prisma = global.prisma || new PrismaClient()


if (process.env.NODE_ENV === 'development') global.prisma = prisma

export default prisma

plus use:

await prisma.$disconnect()

Getting Internal Server Error on prisma deploy

3 Answers3