10

I have a Keycloak installation running as docker container in a docker-compose environment. Every night, my backup stops relevant containers, performs a DB and volume backup and restarts the containers again. For most it works, but Keycloak seems to have a problem with it and does not come up again afterwards. Looking at the logs, the error message is:

The batch failed with the following error: : 
keycloak           | WFLYCTL0062: Composite operation failed and was rolled back. Steps that failed:
keycloak           | Step: step-9
keycloak           | Operation: /subsystem=datasources/jdbc-driver=postgresql:add(driver-name=postgresql, driver-module-name=org.postgresql.jdbc, driver-xa-datasource-class-name=org.postgresql.xa.PGXADataSource)
keycloak           | Failure: WFLYCTL0212: Duplicate resource [
keycloak           |     ("subsystem" => "datasources"),
keycloak           |     ("jdbc-driver" => "postgresql")
keycloak           | ]
...
The batch failed with the following error: : 
keycloak           | WFLYCTL0062: Composite operation failed and was rolled back. Steps that failed:
keycloak           | Step: step-9
keycloak           | Operation: /subsystem=datasources/jdbc-driver=postgresql:add(driver-name=postgresql, driver-module-name=org.postgresql.jdbc, driver-xa-datasource-class-name=org.postgresql.xa.PGXADataSource)
keycloak           | Failure: WFLYCTL0212: Duplicate resource [
keycloak           |     ("subsystem" => "datasources"),
keycloak           |     ("jdbc-driver" => "postgresql")
keycloak           | ]

The docker-compose.yml entry for Keycloak looks as follows, important data obviously removed

  keycloak:
    image: jboss/keycloak:8.0.1
    container_name: keycloak
    environment:
      - PROXY_ADDRESS_FORWARDING=true
      - DB_VENDOR=postgres
      - DB_ADDR=db
      - DB_DATABASE=keycloak
      - DB_USER=keycloak
      - DB_PASSWORD=<password>
      - VIRTUAL_HOST=<url>
      - VIRTUAL_PORT=8080
      - LETSENCRYPT_HOST=<url>
    volumes:
      - /opt/docker/keycloak-startup:/opt/jboss/startup-scripts

The volume I'm mapping is there to make some changes to WildFly to make sure it behaves well with the reverse proxy:

embed-server --std-out=echo

#  Enable https listener for the new security realm
/subsystem=undertow/ \
  server=default-server/ \
    http-listener=default \
      :write-attribute(name=proxy-address-forwarding, \
                       value=true)

#  Create new socket binding with proxy https port
/socket-binding-group=standard-sockets/ \
  socket-binding=proxy-https \
    :add(port=443)

#  Enable https listener for the new security realm
/subsystem=undertow/ \
  server=default-server/ \
    http-listener=default \
      :write-attribute(name=redirect-socket, \
                       value="proxy-https")

After stopping the container, its not starting anymore with the messages shown above. Removing the container and re-creating it works fine however. I tried to remove the volume after the initial start, this doesn't really make a difference either. I already learned that I have to remove the KEYCLOAK_USER=admin and KEYCLOAK_PASSWORD environment variables after the initial boot as otherwise the container complains that the user already exists and doesn't start anymore. Any idea how to fix that?

Daniel
  • 1,398
  • 4
  • 20
  • 47

3 Answers3

9

Update on 23rd of May 2021:

The issue has been resolved on RedHats Jira, it seems to be resolved in version 12. The related GitHub pull request can be found here: https://github.com/keycloak/keycloak-containers/pull/286

According to RedHat support, this is a known "issue" and not supposed to be fixed. They want to concentrate on a workflow where a container is removed and recreated, not started and stopped. They agreed with the general problem, but stated that currently there are no resources available. Stopping and starting the container is a operation which is currently not supported.

See for example https://issues.redhat.com/browse/KEYCLOAK-13094?jql=project%20%3D%20KEYCLOAK%20AND%20text%20~%20%22docker%20restart%22 for reference

Daniel
  • 1,398
  • 4
  • 20
  • 47
2

A legitimate use case for restarting is to add debug logging. For example to debug authentication with an external identity provider.

I ended up creating a shell script that does:

  • docker stop [container]
  • docker rm [container]
  • recreates the image i want with changes to the logging configuration
  • docker run [options] [container]

However a nice feature of docker is the ability to restart a stopped container automatically, decreasing downtime. This Keycloak bug takes that feature away.

Robin
  • 612
  • 1
  • 5
  • 12
-1

I had the same problem here, and my solution was:

  1. Export docker container to a .tar file:

docker export CONTAINER_NAME > latest.tar

2- Create a new volume in a docker

docker volume create VOLUME_NAME

3 - Start a new docker container mapping the volume created to a container db path, something like this:

docker run --name keycloak2 -v keycloak_db:/opt/jboss/keycloak/standalone/data/ -p 8080:8080 -e PROXY_ADDRESS_FORWARDING=true -e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=root jboss/keycloak

4 - Stop the container

5 - Unpack the tar file and find the database path, something like this:

tar unpack path: /opt/jboss/keycloak/standalone/data

6 - Move the path content to docker volume, if you dont know where is the physical path use docker inspect volume VOLUME_NAME to find the path

7 - Start the stoped container

This works for me, I hope its so helpfull to the next person to fix this problem.

Mauro Grabowski Junior
  • 1