Getting Secure Kafka and Schema registry to talk to each other, as well as spring boot configuration for it

Question

Hey I have been trying to solve this for about two weeks now. Basically I want kafka to be on SSL and schema registry on HTTPS. No kerberos to be used. I have a two spring services, one is a producer and the other a consumer (avro)

this is my current docker-compose, with it when I send a request to my producer it doesn't throw any errors in the application, the request times out but Kafka logs show kafka_1 | [2019-12-03 09:53:27,454] INFO [SocketServer brokerId=1] Failed authentication with /172.18.0.1 (SSL handshake failed) (org.apache.kafka.common.network.Selector) when I uncomment the lines in the docker-compose i get PKIX path building failed and some other error specifying that Avro cannot be serialized or something like that

  zookeeper:
    image: confluentinc/cp-zookeeper:5.3.0
    ports:
      - 2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: "2181"
      ZOOKEEPER_TICK_TIME: "2000"

  kafka:
    image: confluentinc/cp-kafka:5.3.0
    ports:
      - 29094:29094

    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,SSL:SSL
      KAFKA_SECURITY_PROTOCOL: SSL
      KAFKA_INTER_BROKER_PROTOCOL: SSL
      KAFKA_INTER_BROKER_LISTENER_NAME: SSL
      KAFKA_LISTENERS: SSL://kafka:29094,PLAINTEXT://kafka:9092
      KAFKA_ADVERTISED_LISTENERS: SSL://kafka:29094,PLAINTEXT://kafka:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
#      KAFKA_SSL_CLIENT_AUTH: required
      KAFKA_SSL_KEYSTORE_FILENAME: kafka.server.keystore.jks
      KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
      KAFKA_SSL_KEY_CREDENTIALS: key_credential
      KAFKA_SSL_KEYSTORE_CREDENTIALS: key_credential
      KAFKA_SSL_KEYSTORE_LOCATION: /etc/kafka/secrets/kafka.server.keystore.jks
      KAFKA_SSL_KEYSTORE_PASSWORD: PASSWORD
      KAFKA_SSL_KEY_PASSWORD: PASSWORD
      KAFKA_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/kafka.server.truststore.jks
      KAFKA_SSL_TRUSTSTORE_PASSWORD: PASSWORD
      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: key_credential
      KAFKA_HEAP_OPTS: -Xmx456M
      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
      #            KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.auth.SimpleAclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
      KAFKA_SUPER_USERS: User:CN=Kafka-domain

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /home/xubu/Documents:/etc/kafka/secrets


  schema-registry:
    image: confluentinc/cp-schema-registry:5.3.0
    depends_on:
      - zookeeper
      - kafka
    ports:
      - 8181:8181
      - 8085:8085
      - 8086:8086
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_LISTENERS: http://schema-registry:8085, https://schema-registry:8086

      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181
      SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL: SSL
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: SSL://kafka:29094
      SCHEMA_REGISTRY_KAFKASTORE_SSL_TRUSTSTORE_LOCATION: /etc/kafka/client/kafka.client.truststore.jks
      SCHEMA_REGISTRY_KAFKASTORE_SSL_TRUSTSTORE_PASSWORD: PASSWORD
      SCHEMA_REGISTRY_KAFKASTORE_SSL_KEYSTORE_LOCATION: /etc/kafka/client/kafka.client.keystore.jks
      SCHEMA_REGISTRY_KAFKASTORE_SSL_KEYSTORE_PASSWORD: PASSWORD
      SCHEMA_REGISTRY_KAFKASTORE_SSL_KEY_PASSWORD: PASSWORD
      SCHEMA_REGISTRY_KAFKASTORE_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""

      SCHEMA_REGISTRY_SSL_TRUSTSTORE_LOCATION: /etc/kafka/client/kafka.client.truststore.jks
      SCHEMA_REGISTRY_SSL_TRUSTSTORE_PASSWORD: PASSWORD
      SCHEMA_REGISTRY_SSL_KEYSTORE_LOCATION: /etc/kafka/client/kafka.client.keystore.jks
      SCHEMA_REGISTRY_SSL_KEYSTORE_PASSWORD: PASSWORD
      SCHEMA_REGISTRY_SSL_KEY_PASSWORD: PASSWORD
      SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""
      SCHEMA_REGISTRY_SECURITY_PROTOCOL: SSL
      SCHEMA_REGISTRY_INTER_INSTANCE_PROTOCOL: "https"

      #SCHEMA_REGISTRY_SSL_CLIENT_AUTH: 'true'



    volumes:
      - /home/xubu/Documents:/etc/kafka/client
      - /home/xubu/Documents:/etc/kafka/consumer

Below is part of my spring boot application.yaml

spring:
kafka:
    bootstrap-servers: kafka:29094
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer

      ssl:
        key-store-location: /home/xubu/Documents/kafka.client.keystore.jks
        key-password: PASSWORD
        key-store-password: PASSWORD
        trust-store-location: /home/xubu/Documents/kafka.client.truststore.jks
        trust-store-password: PASSWORD
        protocol: SSL


    properties:
      value:
        subject:
          name:
            strategy: io.confluent.kafka.serializers.subject.RecordNameStrategy
      value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
      ssl.endpoint.identification.algorithm: https
      schema.registry.url: https://schema-registry:8086
    ssl:
      trust-store-location: /home/xubu/Documents/kafka.client.truststore.jks
      trust-store-password: PASSWORD
      key-store-location: /home/xubu/Documents/kafka.client.keystore.jks
      key-store-password: PASSWORD
      key-password: PASSWORD
      protocol: SSL
      key-store-type: jks
      trust-store-type: jks

This has been my pain for the last two or so weeks, and it's an intro into trying out Access Control Lists on schema-registry