0

I am new to kubectl and helm charts but I am using both to deploy Fortify SSC to a k8s cluster. The k8s resources and the application seem to deploy correctly from an iac perspective but I keep getting java.io.IOException: Invalid keystore format errors in the application logs:

**Output: ** kubectl logs foo-bar-0 --all-containers

...
26-May-2023 18:57:13.174 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-jsse-nio2-8443"]
26-May-2023 18:57:18.377 WARNING [main] org.apache.tomcat.util.net.Nio2Endpoint.shutdownExecutor The executor associated with thread pool [https-jsse-nio2-8443] has not fully shutdown. Some application threads may still be running.    
26-May-2023 18:57:18.378 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[org.apache.coyote.http11.Http11Nio2Protocol-8443]]
        org.apache.catalina.LifecycleException: Protocol handler initialization failed
                at org.apache.catalina.connector.Connector.initInternal(Connector.java:1013)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
                at org.apache.catalina.core.StandardService.initInternal(StandardService.java:533)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
                at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1057)
                at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
                at org.apache.catalina.startup.Catalina.load(Catalina.java:584)
                at org.apache.catalina.startup.Catalina.load(Catalina.java:607)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:303)
                at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
        **Caused by: java.lang.IllegalArgumentException: Invalid keystore format**
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:99)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71)
                at org.apache.tomcat.util.net.Nio2Endpoint.bind(Nio2Endpoint.java:142)
                at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1141)
                at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1154)
                at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581)
                at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:74)
                at org.apache.catalina.connector.Connector.initInternal(Connector.java:1010)
                ... 13 more
        **Caused by: java.io.IOException: Invalid keystore format**
                at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:663)
                at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
                at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
                at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
                at java.security.KeyStore.load(KeyStore.java:1445)
                at org.apache.tomcat.util.security.KeyStoreUtil.load(KeyStoreUtil.java:69)
                at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:217)
                at org.apache.tomcat.util.net.SSLHostConfigCertificate.getCertificateKeystore(SSLHostConfigCertificate.java:206)
                at org.apache.tomcat.util.net.SSLUtilBase.getKeyManagers(SSLUtilBase.java:283)
                at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:247)
                at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:97)
                ... 20 more
26-May-2023 18:57:18.379 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [5,451] milliseconds
...

I'm not sure which parts of the helm/k8s configs are relevant here but I initially deployed the configuration with:

helm folder

charts/fortify/
├── Chart.yaml
├── secrets
│   ├── keystore-pass
│   ├── keystore.jks
│   └── autoconfig.yaml
├── foobar.tgz
└── foo-values.yaml

foo-values.yml

# Custom values for foobar.tgz chart.
# This is a YAML-formatted file.
...
secretRef:
  name: "my-secrets"
  keys:
...
    httpCertificateKeystoreFileEntry: "keystore.jks"

...
volumes:
  - name: my-secrets
    secret:
      secretName: my-secrets
      defaultMode: 420
...

volumeMounts:
  - name: my-secrets
    mountPath: "/mounted/path/to/folder"

...

Commands

kubectl create secret generic my-secrets --from-file=my-secrets-folder
helm install my-app foobar.tgz --values foo-values.yaml

I tracked the issue down to the keystore.jks (mounted) file. Somehow the file is getting corrupted during kubectl commands?

When I run keytool -list -v -keystore keystore.jks on the host I get the normal keytool output:

root@host:/# keytool -list -v -keystore path/to/keystore.jks 
Enter keystore password:  
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: example.com
Creation date: May 22, 2023
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
...

When I run the same command within the pod/container (kubectl exec --stdin --tty foo-bar-0 -- /bin/bash) I get:

root@foo-bar-0:/pvc# keytool -list -v -keystore /mounted/path/to/keystore.jks
Enter keystore password:  
keytool error: java.io.IOException: Invalid keystore format
java.io.IOException: Invalid keystore format
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:663)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at sun.security.tools.keytool.Main.doCommands(Main.java:926)
        at sun.security.tools.keytool.Main.run(Main.java:366)
        at sun.security.tools.keytool.Main.main(Main.java:359)
root@foo-bar-0:/pvc#

I also tried several different ways to create the secrets resource with kubectl apply -f secrets_v#.yaml. Each secrets_v#.yaml file shown is a slightly different way to accomplish creating the same secret resource.

secrets_v1.yaml

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: my-secrets
  namespace: default
data:
...
  keystore.jks: <base64 encoded string>
...

secrets_v2.yaml

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: my-secrets
data:
...
  keystore.jks: {{ .Files.Get "keystore.jks" | b64enc }}
...

secrets_v3.yaml

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: my-secrets
stringData:
  keystore.jks: |
    "<base64 encoded string>"

secrets_v4.yaml

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: my-secrets
stringData:
  keystore.jks: |-
    {{ .Files.Get "keystore.jks" | b64enc }}

EDIT

kubectl get secret ssc-secrets -o yaml:

apiVersion: v1
data:
  keystore-pass: <base64 encoded string>
  keystore.jks: <base64 encoded string>
  autoconfig.yaml: <base64 encoded string>
kind: Secret
metadata:
  creationTimestamp: "2023-05-26T18:43:04Z"
  name: my-secrets
  namespace: default
  resourceVersion: "13362"
  uid: 3fac5611-aa76-4208-9a09-33ad55d5acf0
type: Opaque

kubectl describe secret my-secrets

Name:         my-secrets
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
keystore-pass:        9 bytes
keystore.jks:         1246 bytes
autoconfig.yaml:      578 bytes
John R
  • 350
  • 2
  • 5
  • 19
  • Did you try to "cat" the keystore from the pod? Happened with me that the file is there but it was empty, because of syntax error on mounting. Also how you mount it is a bit confusing in the given example under the "foo-values.yml". Maybe you could elaborate on that more since that is causing the problem probably. – adam sranko May 26 '23 at 22:45
  • cat gives me the "expected" binary file output (at least we know its not empty). root@foo-bar-0:/# cat /mounted/path/to/keystore.jks ��%�!0�0��� *�H�� *�H�� 080��*�H�� *�H�� 0+1�Rx�o ��B���A�Xa�' 0 0 `�He*38 #�gCt�IW{T ��PJ�܈ze�0ĶCV��L��r���Ϯ G>��^� 08"0�*�H��� 168480*�H��860�1� *�H�� ��c�y���*��x-9?��y9[��璙��?V�\�� – John R May 26 '23 at 22:51
  • Well that means the content of the kubernetes secret is broken/malformatted I think. You can do "kubectl get secret yoursecretname -o yaml" maybe add that here with some part cut out of it or just check for yourself if it looks fine. – adam sranko May 26 '23 at 22:55
  • cat on a java key store will normally have binary output like the above (the same also happens on the host where keytool works correctly). – John R May 26 '23 at 23:47
  • I did not mean that its "broken" because it is binary it can just contain unwanted breaklines etc which ultimately breaks it. If you would compare the mounted keystore and the original byte by byte I assume it would not be the exact same. – adam sranko May 27 '23 at 10:09

0 Answers0