Ensuring spring boot and liquibase receive and handle SIGTERM

Question

Currently running SpringBoot applications in a containerised environment (ECS) and I've observed scenarios in which the container gets terminated during start-up and while it's still holding the Liquibase changelock.

This leads to issues in all containers that are spun afterwards and ends up requiring manual intervention.

Is it possible to ensure that if the process receives a SIGTERM, it will gracefully handle termination and release the lock?

I've already ensured that the container is receiving the signals by enabling via InitProcessEnabled (in the CloudFormation template) and use of "exec java ..." as a java agent we use does gracefully shutdown on this circumstances.

Did you find a solution for the problem? We're running in the same problem with our App running in a Wildfly. — Marvin, Aug 10 '20 at 10:01
Create an issue in the Liqubase repo now: https://github.com/liquibase/liquibase/issues/1311 — Marvin, Aug 11 '20 at 07:29
No unfortunately I didn't find a solution for this problem we've disabled liquibase and took another approach. Personally i'm still quite curious to see how it goes (and I guess I shoulld've raised an issue in their repo). — bayetovsky, Aug 11 '20 at 12:57
@bayetovsky Have a look here https://stackoverflow.com/a/65564995/1704634 . Using the liquibase-sessionlock extension it will automatically release the lock as soon as the container is terminated. — blagerweij, Jan 12 '21 at 10:16

Christoph Pelzer · Accepted Answer · 2021-02-15T23:13:33.230

Heyo,

As mentioned in the GitHub issue I have a workaround. A solution is yet to be implemented.

You can manually register a shutdown hook before running spring boot.. That hook should assure that the Termination is postponed until liquibase is done.

package dang;

import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;


@EnableJpaRepositories
@SpringBootApplication
public class DangApplication {
  public static void main(String[] args) throws InterruptedException {
    Thread thread = new GracefulShutdownHook();
    Runtime.getRuntime().addShutdownHook(thread);

    new SpringApplicationBuilder(DangApplication.class)
            .registerShutdownHook(true)
            .logStartupInfo(true)
            .build()
            .run();
    Runtime.getRuntime().removeShutdownHook(thread);
  }
}

And the hook:

package dang;

import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;

import java.util.Map;

@Slf4j
public class GracefulShutdownHook extends Thread {
  @SneakyThrows
  @Override
  public void run() {


    super.run();
    log.info("Shutdown Signal received.. Searching for Liquibase instances!");
    boolean liquibaseIsRunning = true;
    while (liquibaseIsRunning) {

      Map<Thread,StackTraceElement[]> stackTraces = Thread.getAllStackTraces();
      for(Map.Entry<Thread, StackTraceElement[]> entry : stackTraces.entrySet()) {
        StackTraceElement[] stackTraceElements = entry.getValue();
        for (StackTraceElement stackTraceElement : stackTraceElements) {
          if (stackTraceElement.getClassName().contains("liquibase") && stackTraceElement.getMethodName().contains("update")) {
            try {
              log.warn("Liquibase is currently updating");
              entry.getKey().join();
              liquibaseIsRunning = false;
            } catch (InterruptedException e) {
              log.error("Shutdown Hook was interrupted.. Fatal databaselock may be imminent", e);
              if (Thread.interrupted()) {
                throw e;
              }
            }
          }
        }
      }
    }
  }
}

EDIT

After implementing my workaround a contributor of liquibase shared a different solution (It's actually the same solution just through Spring functionality) which is much better than what I did:

package dang;

import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;


@EnableJpaRepositories
@SpringBootApplication
public class DangApplication {
  public static void main(String[] args) throws InterruptedException {
    new SpringApplicationBuilder(DangApplication.class)
            .initializers(ConfigurableApplicationContext::registerShutdownHook) // Registers application hook before liquibase executes.
            .logStartupInfo(true)
            .build()
            .run();
  }
}

thanks for sharing that info! accepted this one as the answer as it is indeed the most helpful (and closest to solving the problem). Should also be noted that the comment that @blagerweij did pointing to https://stackoverflow.com/a/65564995/1704634 is also a very good answer and possible solution to this problem. Hopefully this helps anyone that experiences similar problems — bayetovsky, Jan 13 '21 at 11:34
Heyo @bayetovsky thanks for accepting my answer.. After sharing my workaround on the liquibase github page a contributor suggested doing the same thing just with the SpringBoot shutdown hook.. Look at my edit and you see a huge difference (no GracefulShutdown class needed :D ) — Christoph Pelzer, Feb 15 '21 at 23:14

Ensuring spring boot and liquibase receive and handle SIGTERM

1 Answers1