I'm implementing graceful termination for a Golang Application deployed on a Kubernetes cluster. I'm using pgxconn.Pool.
My main challenge now is to forcefully kill all queries from the Application that are running on the PostgreSQL server during shutdown.
The Deployment on Kubernetes is defined with terminationGracePeriodSeconds: 30 (default), so unless I kill all queries within 30 seconds the pod will be killed by Kubernetes and the queries will keep running until completion with no one on the other side waiting for the result. This impacts the performance and stability of the database.
In my code I'm catching termination signals and calling Shutdown(ctx context.Context) on my http.Server with a timeout. Once the server has been closed, I'm calling Close() on pgxconn.Pool, but the Close function doesn't have take a context and I don't see any other way to forcefully kill all queries and close connections.
Here's the code in my main() function:
// The sum of connPoolCloseTimeout and serverCloseTimeout must be less than 30
// seconds - the time Kubernetes will wait after a SIGTERM before it kills the pod
// (terminationGracePeriodSeconds)
const connPoolCloseTimeout, serverCloseTimeout time.Duration = time.Second*10, time.Second*10
go func() {
signalChan := make(chan os.Signal, 1)
signal.Notify(signalChan,
syscall.SIGTERM,
syscall.SIGQUIT,
syscall.SIGINT,
)
sig := <-signalChan
mainLogger.Warn("got signal", zap.String("signal", sig.String()))
ctx, cancel := context.WithTimeout(context.Background(), serverCloseTimeout)
defer cancel()
if err := server.Shutdown(ctx); err != nil && err != http.ErrServerClosed {
mainLogger.Warn("server Shutdown (forced shutdown due to timeout):", zap.Error(err))
} else if err != nil {
mainLogger.Warn("server Shutdown (error):", zap.Error(err))
} else {
mainLogger.Info("server Shutdown (clean)")
}
//Close waits for all acquired connections to be released
//Because of this we must add a deadline or this will block forever
doneChan := make(chan struct{}, 1)
go func() {
mainLogger.Info("Closing connection pool. Waiting for all connections to close")
pool.Close()
doneChan <- struct{}{}
}()
select {
case <-doneChan:
//Do nothing and continue
case <-time.After(connPoolCloseTimeout):
dbc.logger.Error("closing connection pool by force, deadline exceeded")
// TODO: In this case, we couldn't manage to close the connection pool in time. The application will be terminated and ongoing queries will not be killed. If there's a method to force kill queries add it here
}
}()
Is there a good way (without manually sending RST packets to the PostgreSQL server or similar hacks) to forcefully close all running queries from a connection pool upon termination?
Will I have to execute each query with a Context and call the cancel function of every such context on shutdown, or will this also not help?
[I already have a timeout set for every DB query, but that timeout is rather high to allow long running queries to complete in a normal state of affairs]
