I deployed keycloak to our k8s cluster, with the production start option, but the istio healthchecks and the routing from the virtualservices are running into issues with the specified port. The port the keycloak should be working on is 8443, when starting with the "start" on a production level. Sadly I'm running into the issue that I can only reach the application on some nodeport, which is written out in the logs. Here are the config files and logs for the application.
Keycloak Logs:
2023-04-14 08:45:18,577 INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: keycloak-sandbox.v2docusketch.com, Strict HTTPS: true, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: false
2023-04-14 08:45:20,627 WARN [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
2023-04-14 08:45:21,954 INFO [org.infinispan.SERVER] (keycloak-cache-init) ISPN005054: Native IOUring transport not available, using NIO instead: io.netty.incubator.channel.uring.IOUring
2023-04-14 08:45:22,142 WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2023-04-14 08:45:22,284 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2023-04-14 08:45:22,345 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2023-04-14 08:45:23,146 INFO [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2023-04-14 08:45:23,292 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2023-04-14 08:45:23,537 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2023-04-14 08:45:23,563 INFO [org.jgroups.JChannel] (keycloak-cache-init) local_addr: 8630de85-2011-43d7-a9e9-73729d8ea75a, name: keycloak-657b749c64-22gxr-61765
2023-04-14 08:45:23,588 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
2023-04-14 08:45:23,589 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20MB, but the OS only allocated 212.99KB
2023-04-14 08:45:23,589 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
2023-04-14 08:45:23,589 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB
2023-04-14 08:45:23,600 INFO [org.jgroups.protocols.FD_SOCK2] (keycloak-cache-init) server listening on *.27075
2023-04-14 08:45:25,642 INFO [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) keycloak-657b749c64-22gxr-61765: no members discovered after 2024 ms: creating cluster as coordinator
2023-04-14 08:45:25,652 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [keycloak-657b749c64-22gxr-61765|0] (1) [keycloak-657b749c64-22gxr-61765]
2023-04-14 08:45:25,658 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `keycloak-657b749c64-22gxr-61765`, physical addresses are `[10.0.21.39:42611]`
2023-04-14 08:45:26,340 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak-657b749c64-22gxr-61765, Site name: null
2023-04-14 08:45:27,584 INFO [io.quarkus] (main) Keycloak 21.0.2 on JVM (powered by Quarkus 2.13.7.Final) started in 11.309s. Listening on: https://0.0.0.0:3000
2023-04-14 08:45:27,585 INFO [io.quarkus] (main) Profile prod activated.
2023-04-14 08:45:27,585 INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy, resteasy-jackson, smallrye-context-propagation, smallrye-health, vertx]
2023-04-14 08:45:27,595 ERROR [org.keycloak.services] (main) KC-SERVICES0010: Failed to add user 'admin' to realm 'master': user with username exists
Istio Logs:
2023-04-14T08:45:16.291192Z info FLAG: --concurrency="2"
2023-04-14T08:45:16.291389Z info FLAG: --domain="ms-docusketch-keycloak.svc.cluster.local"
2023-04-14T08:45:16.291504Z info FLAG: --help="false"
2023-04-14T08:45:16.291570Z info FLAG: --log_as_json="false"
2023-04-14T08:45:16.291627Z info FLAG: --log_caller=""
2023-04-14T08:45:16.291729Z info FLAG: --log_output_level="default:info"
2023-04-14T08:45:16.291795Z info FLAG: --log_rotate=""
2023-04-14T08:45:16.291847Z info FLAG: --log_rotate_max_age="30"
2023-04-14T08:45:16.291920Z info FLAG: --log_rotate_max_backups="1000"
2023-04-14T08:45:16.291972Z info FLAG: --log_rotate_max_size="104857600"
2023-04-14T08:45:16.292023Z info FLAG: --log_stacktrace_level="default:none"
2023-04-14T08:45:16.292117Z info FLAG: --log_target="\[stdout\]"
2023-04-14T08:45:16.292191Z info FLAG: --meshConfig="./etc/istio/config/mesh"
2023-04-14T08:45:16.292242Z info FLAG: --outlierLogPath=""
2023-04-14T08:45:16.292319Z info FLAG: --proxyComponentLogLevel="misc:error"
2023-04-14T08:45:16.292362Z info FLAG: --proxyLogLevel="warning"
2023-04-14T08:45:16.292433Z info FLAG: --serviceCluster="istio-proxy"
2023-04-14T08:45:16.292483Z info FLAG: --stsPort="0"
2023-04-14T08:45:16.292559Z info FLAG: --templateFile=""
2023-04-14T08:45:16.292601Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2023-04-14T08:45:16.292673Z info FLAG: --vklog="0"
2023-04-14T08:45:16.292722Z info Version 1.16.1-f6d7bf648e571a6a523210d97bde8b489250354b-Clean
2023-04-14T08:45:16.295030Z info Maximum file descriptors (ulimit -n): 1048576
2023-04-14T08:45:16.295417Z info Proxy role ips=\[10.0.21.39\] type=sidecar id=keycloak-657b749c64-22gxr.ms-docusketch-keycloak domain=ms-docusketch-keycloak.svc.cluster.local
2023-04-14T08:45:16.295588Z info Apply proxy config from env {}
2023-04-14T08:45:16.302118Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
zipkin:
address: zipkin.istio-system:9411
2023-04-14T08:45:16.302143Z info JWT policy is third-party-jwt
2023-04-14T08:45:16.302148Z info using credential fetcher of JWT type in cluster.local trust domain
2023-04-14T08:45:16.303519Z info platform detected is AWS
2023-04-14T08:45:16.305570Z info Workload SDS socket not found. Starting Istio SDS Server
2023-04-14T08:45:16.305888Z info CA Endpoint istiod.istio-system.svc:15012, provider Citadel
2023-04-14T08:45:16.306004Z info Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2023-04-14T08:45:16.306178Z info citadelclient Citadel client using custom root cert: var/run/secrets/istio/root-cert.pem
2023-04-14T08:45:16.305700Z info Opening status port 15020
2023-04-14T08:45:16.326728Z info ads All caches have been synced up in 38.673706ms, marking server ready
2023-04-14T08:45:16.327195Z info xdsproxy Initializing with upstream address "istiod.istio-system.svc:15012" and cluster "Kubernetes"
2023-04-14T08:45:16.329390Z info Pilot SAN: \[istiod.istio-system.svc\]
2023-04-14T08:45:16.331357Z info Starting proxy agent
2023-04-14T08:45:16.331475Z info starting
2023-04-14T08:45:16.331601Z info Envoy command: \[-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --parent-shutdown-time-s 60 --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --log-format %Y-%m-%dT%T.%fZ %l envoy %n %v -l warning --component-log-level misc:error --concurrency 2\]
2023-04-14T08:45:16.335696Z info sds Starting SDS grpc server
2023-04-14T08:45:16.335822Z info starting Http service at 127.0.0.1:15004
2023-04-14T08:45:16.457541Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-04-14T08:45:16.497161Z info ads ADS: new connection for node:keycloak-657b749c64-22gxr.ms-docusketch-keycloak-1
2023-04-14T08:45:16.498484Z info ads ADS: new connection for node:keycloak-657b749c64-22gxr.ms-docusketch-keycloak-2
2023-04-14T08:45:16.662192Z info cache generated new workload certificate latency=329.50278ms ttl=23h59m59.337822618s
2023-04-14T08:45:16.662395Z info cache Root cert has changed, start rotating root cert
2023-04-14T08:45:16.662507Z info ads XDS: Incremental Pushing:0 ConnectedEndpoints:2 Version:
2023-04-14T08:45:16.662656Z info cache returned workload trust anchor from cache ttl=23h59m59.337368268s
2023-04-14T08:45:16.662819Z info cache returned workload trust anchor from cache ttl=23h59m59.337185287s
2023-04-14T08:45:16.663326Z info ads SDS: PUSH request for node:keycloak-657b749c64-22gxr.ms-docusketch-keycloak resources:1 size:1.1kB resource:ROOTCA
2023-04-14T08:45:16.663688Z info cache returned workload trust anchor from cache ttl=23h59m59.336319111s
2023-04-14T08:45:16.663860Z info cache returned workload certificate from cache ttl=23h59m59.336144565s
2023-04-14T08:45:16.664050Z info ads SDS: PUSH request for node:keycloak-657b749c64-22gxr.ms-docusketch-keycloak resources:1 size:4.0kB resource:default
2023-04-14T08:45:17.004048Z error Request to probe app failed: Get "https://10.0.21.39:3000/health": dial tcp 127.0.0.6:0-\>10.0.21.39:3000: connect: connection refused, original URL path = /app-health/keycloak/readyz
app URL path = /health
2023-04-14T08:45:17.008949Z info Readiness succeeded in 729.33108ms
2023-04-14T08:45:17.009515Z info Envoy proxy is ready
2023-04-14T08:45:18.044091Z error Request to probe app failed: Get "https://10.0.21.39:3000/health": dial tcp 127.0.0.6:0-\>10.0.21.39:3000: connect: connection refused, original URL path = /app-health/keycloak/readyz
app URL path = /health
2023-04-14T08:45:19.081692Z error Request to probe app failed: Get "https://10.0.21.39:3000/health": dial tcp 127.0.0.6:0-\>10.0.21.39:3000: connect: connection refused, original URL path = /app-health/keycloak/readyz
app URL path = /health
2023-04-14T08:45:24.505026Z error Request to probe app failed: Get "https://10.0.21.39:3000/health": dial tcp 127.0.0.6:0-\>10.0.21.39:3000: connect: connection refused, original URL path = /app-health/keycloak/readyz
app URL path = /health
2023-04-14T08:45:24.505118Z error Request to probe app failed: Get "https://10.0.21.39:3000/health": dial tcp 127.0.0.6:0-\>10.0.21.39:3000: connect: connection refused, original URL path = /app-health/keycloak/livez
app URL path = /health
2023-04-14T09:16:52.618264Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-04-14T09:46:41.562970Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-04-14T10:15:11.889381Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-04-14T10:46:43.879346Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-04-14T11:19:31.724799Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-04-14T11:48:10.600561Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
2023-04-14T12:19:20.542636Z info xdsproxy connected to upstream XDS server: istiod.istio-system.svc:15012
Deployment.yaml
ports:
- name: https
containerPort: 8443
protocol: TCP
livenessProbe:
httpGet:
path: /health
port: https
scheme: HTTPS
readinessProbe:
httpGet:
path: /health
port: https
scheme: HTTPS
env:
- name: KEYCLOAK_ADMIN
value: admin
- name: KEYCLOAK_ADMIN_PASSWORD
value: ***
- name: KC_DB
value: postgres
- name: KC_DB_SCHEMA
value: auth
- name: KC_DB_URL
value: jdbc:postgresql://***/keycloak
- name: KC_DB_USERNAME
value: ***
- name: KC_DB_PASSWORD
value: ***
- name: PROXY_ADDRESS_FORWARDING
value: 'true'
args:
- start
- '--optimized'
- '--import-realm'
- '--hostname=keycloak-sandbox.v2docusketch.com'
- '--https-key-store-file=/opt/keycloak/conf/server.keystore'
Dockerfile:
FROM quay.io/keycloak/keycloak:latest as builder
# Enable health and metrics support
ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
# Configure a database vendor
ENV KC_DB=postgres
WORKDIR /opt/keycloak
# for demonstration purposes only, please make sure to use proper certificates in production instead
RUN keytool -genkeypair -storepass password -storetype PKCS12 -keyalg RSA -keysize 2048 -dname "CN=server" -alias server -ext "SAN:c=DNS:localhost,IP:127.0.0.1" -keystore conf/server.keystore
RUN /opt/keycloak/bin/kc.sh build
FROM quay.io/keycloak/keycloak:latest
COPY --from=builder /opt/keycloak/ /opt/keycloak/
ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]
Can someone help me to understand, why the forwarding on port 8443 does not work as expected? And the healthchecks for istio are running into Error 500, did someone experience this as well and can help me fix that issue?
I'm trying to start keycloak on production level and deploy it with https under the port 8443 in the kubernetes cluster. Sadly this is not working as expected and the port 8443 is not serving the application as expected.
