Hazelcast: mancenter serialization configuration ignored

Question

I've created this hazelcast configuration into my mancenter on /config/hazelcast-client.yaml:

hazelcast-client:
  network:
    kubernetes:
      enabled: true
      namespace: ${namespace}
      service-name: ${serviceName}
  serialization:
    serializers:
    - class-name: me.jeusdi.fhir.hazelcast.serializer.PractitionerSerializer
      type-class: me.jeusdi.fhir.hazelcast.model.Practitioner

This is my mancenter log:

$ kubectl logs -f hazelcast-external-mancenter-0

Container support enabled. Using automatic heap sizing. JVM will use up to 80% of the memory limit of the container.
Executing command specified by MC_INIT_CMD for container initialization.
WARNING: Cluster null already exists.

To see the full stack trace, re-run with the -v/--verbose option.
##################################################
# Initialisation complete, starting now....      #
##################################################
+ exec java --add-opens java.base/java.lang=ALL-UNNAMED -server -Dhazelcast.mc.home=/data -Djava.net.preferIPv4Stack=true --illegal-access=permit -Dhazelcast.mc.healthCheck.enable=true -DserviceName=hazelcast-external -Dnamespace=default -Dhazelcast.mc.tls.enabled=false -Dhazelcast.mc.lock.skip=true -XX:+UseContainerSupport -XX:MaxRAMPercentage=80 -cp '/opt/hazelcast/management-center/hazelcast-management-center-5.0.4.jar:/data/custom:/data/custom/*' -Dhazelcast.mc.contextPath=/ -Dhazelcast.mc.http.port=8080 -Dhazelcast.mc.https.port=8443 com.hazelcast.webmonitor.Launcher
2022-03-07 20:54:28,614 [ INFO] [main] [c.h.w.c.BuildInfo]: Hazelcast Management Center 5.0.4 (20211220 - c20d028), Hazelcast client version: 5.0.1, embedded Jetty version: 9.4.43.v20210629
2022-03-07 20:54:28,718 [ INFO] [main] [c.h.w.Launcher]: Health check is enabled and available at http://localhost:8081/health
2022-03-07 20:54:30,830 [ INFO] [main] [c.h.w.c.SqlDbConfig]: Checking DB for required migrations.
2022-03-07 20:54:31,142 [ INFO] [main] [c.h.w.c.SqlDbConfig]: Number of applied DB migrations: 0.
2022-03-07 20:54:31,322 [ INFO] [main] [c.h.w.s.s.i.DisableLoginStrategy]: Login will be disabled for 5 seconds after 3 failed login attempts. For every 3 consecutive failed login attempts, disable period will be multiplied by 10.
2022-03-07 20:54:33,134 [ INFO] [main] [c.h.w.s.ClusterManager]: Connecting to 1 enabled cluster(s) on startup.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/opt/hazelcast/management-center/hazelcast-management-center-5.0.4.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-03-07 20:54:33,955 [ INFO] [MC-Client-dev.lifecycle-1] [c.h.w.s.MCClientManager]: MC Client connected to cluster dev.
2022-03-07 20:54:33,985 [ INFO] [MC-Client-dev.event-4] [c.h.w.s.MCClientManager]: Started communication with member: Member [10.42.1.57]:5701 - 5ecf6666-505c-4c58-a773-7661be585cb6
2022-03-07 20:54:34,209 [ INFO] [main] [c.h.w.Launcher]: Hazelcast Management Center successfully started at http://localhost:8080/
2022-03-07 20:55:13,115 [ INFO] [MC-Client-dev.event-1] [c.h.w.s.MCClientManager]: Started communication with a new member: Member [10.42.1.58]:5701 - 30c83498-06a9-4e88-ad8a-85398bb45279
2022-03-07 20:55:42,929 [ INFO] [MC-Client-dev.event-1] [c.h.w.s.MCClientManager]: Started communication with a new member: Member [10.42.1.59]:5701 - 90f97bd4-47a1-4f18-8811-b11fa54f2008

Problem arises, when I'm trying to explore any Map. I'm getting:

{ "message":"Server request error: Failed to deserialize map entry. Make sure that you use a client config file with relevant serialization configuration for your cluster client config. Exception message: There is no suitable de-serializer for type 1. This exception is likely caused by differences in the serialization configuration between members or between clients and members." }

EDIT:

I've installed hazelcast using official helm chart (chart version: 5.3.4 / app version: 5.0.2).

My helm values file is:

hazelcast:
  yaml:
    hazelcast:
      network:
        join:
          kubernetes:
            enabled: true
            service-name: ${serviceName}
            namespace: ${namespace}
        rest-api:
          enabled: true
      jet:
        enabled: ${hz.jet.enabled}
      serialization:
        serializers:
          - type-class: me.jeusdi.fhir.hazelcast.model.Practitioner
            class-name: me.jeusdi.fhir.hazelcast.serializer.PractitionerSerializer

customVolume:
  persistentVolumeClaim:
    claimName: hazelcast-jars


mancenter:
  enabled: true
  javaOpts: -Dhazelcast.mc.lock.skip=true
  yaml:
    hazelcast-client:
      network:
        kubernetes:
          enabled: true
          service-name: ${serviceName}
          namespace: ${namespace}
      serialization:
        serializers:
        - type-class: me.jeusdi.fhir.hazelcast.model.Practitioner
          class-name: me.jeusdi.fhir.hazelcast.serializer.PractitionerSerializer
  customVolume:
    persistentVolumeClaim:
      claimName: hazelcast-mancenter-jars

service:
  type: LoadBalancer

Deployed mancenter related configmap:

apiVersion: v1
data:
  hazelcast-client.yaml: |-
    hazelcast-client:
      network:
        kubernetes:
          enabled: true
          namespace: ${namespace}
          service-name: ${serviceName}
      serialization:
        serializers:
        - class-name: me.jeusdi.fhir.hazelcast.serializer.PractitionerSerializer
          type-class: me.jeusdi.fhir.hazelcast.model.Practitioner
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: hazelcast-external
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2022-03-07T20:54:23Z"
  labels:
    app.kubernetes.io/instance: hazelcast-external
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: hazelcast-mancenter
    helm.sh/chart: hazelcast-5.3.4
  name: hazelcast-external-mancenter-configuration
  namespace: default
  resourceVersion: "297666"
  uid: ea94618f-d890-4158-b716-3c88abea8d1c

This is the content of /config/hazelcast-client.yaml:

hazelcast-client:
  network:
    kubernetes:
      enabled: true
      namespace: ${namespace}
      service-name: ${serviceName}
  serialization:
    serializers:
    - class-name: me.jeusdi.fhir.hazelcast.serializer.PractitionerSerializer
      type-class: me.jeusdi.fhir.hazelcast.model.Practitioner

Launch command is:

java --add-opens java.base/java.lang=ALL-UNNAMED -server -Dhazelcast.mc.home=/data -Djava.net.preferIPv4Stack=true --illegal-access=permit -Dhazelcast.mc.healthCheck.enable=true -DserviceName=hazelcast-external -Dnamespace=default -Dhazelcast.mc.tls.enabled=false -Dhazelcast.mc.lock.skip=true -XX:+UseContainerSupport -XX:MaxRAMPercentage=80 -cp '/opt/hazelcast/management-center/hazelcast-management-center-5.0.4.jar:/data/custom:/data/custom/*' -Dhazelcast.mc.contextPath=/ -Dhazelcast.mc.http.port=8080 -Dhazelcast.mc.https.port=8443 com.hazelcast.webmonitor.Launcher

The content of /data/custom:

$ kubectl exec -ti hazelcast-external-mancenter-0 -- ls /data/custom
fhir-hazelcast-0.0.1-SNAPSHOT.jar

jar content is:

     0 Mon Mar 07 12:28:02 CET 2022 META-INF/
    81 Mon Mar 07 12:28:02 CET 2022 META-INF/MANIFEST.MF
     0 Mon Mar 07 12:28:02 CET 2022 me/
     0 Mon Mar 07 12:28:02 CET 2022 me/jeusdi/
     0 Mon Mar 07 12:28:02 CET 2022 me/jeusdi/fhir/
     0 Mon Mar 07 12:28:02 CET 2022 me/jeusdi/fhir/hazelcast/
     0 Mon Mar 07 12:28:02 CET 2022 me/jeusdi/fhir/hazelcast/serializer/
     0 Mon Mar 07 12:28:02 CET 2022 me/jeusdi/fhir/hazelcast/model/
     0 Mon Mar 07 12:28:02 CET 2022 META-INF/maven/
     0 Mon Mar 07 12:28:02 CET 2022 META-INF/maven/me.jeusdi.fhir.hazelcast/
     0 Mon Mar 07 12:28:02 CET 2022 META-INF/maven/me.jeusdi.fhir.hazelcast/fhir-hazelcast/
  2485 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/serializer/SerializerIdEnum.class
  2480 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/serializer/PractitionerSerializer.class
   904 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/PractitionerRoleHistory.class
   906 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/BackboneElement.class
  3350 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QBackboneElement.class
  1677 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/DomainResource$DomainResourceBuilderImpl.class
  3831 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Address.class
  4543 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Patient$ContactComponent.class
   438 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/BooleanType.class
   556 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/EnumerationType.class
   859 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/PatientHistory.class
  4501 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QPractitionerRoleHistory.class
  1173 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Resource$ResourceBuilderImpl.class
  3799 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/HumanName.class
  5304 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QOrganizationHistory.class
  4389 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QQualification.class
  4677 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QSearchParameter.class
  3008 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QCodeType.class
  4639 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QAddress.class
  4300 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QExtension.class
  3679 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QDomainResource.class
  2855 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Extension.class
  4337 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QIdentifier.class
  1523 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Reference.class
  1622 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/ContactPoint$ContactPointSystem.class
  3427 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QElement.class
  4327 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QMeta.class
  3683 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QReference.class
   441 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/DateTimeType.class
  4627 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QOrganization_ContactComponent.class
   336 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/CodeType.class
  2476 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Meta.class
   358 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/PositiveIntType.class
  2445 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/DomainResource.class
  7409 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QPatient.class
  1870 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/PractitionerRole$PractitionerReference.class
   425 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/UriType.class
  5174 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QPractitionerHistory.class
  3409 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QEnumerationType.class
  2360 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Qualification.class
  6391 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QPractitioner.class
  3489 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QPeriod.class
  3119 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QStringType.class
  6349 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QOrganization.class
  4842 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Resource.class
  3856 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QCodeableConcept.class
   321 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Type.class
   434 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/StringType.class
  1483 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Enumerations$AdministrativeGender.class
  1016 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Period.class
  2027 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Practitioner$PatientReference.class
  2177 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Coding.class
  2592 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Identifier.class
  2325 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/PractitionerRole$PractitionerReference$PractitionerReferenceBuilder.class
  2037 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Practitioner$PractitionerBuilderImpl.class
  1495 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/ContactPoint$ContactPointUse.class
  2513 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Patient$GeneralPractitionerReference$GeneralPractitionerReferenceBuilder.class
  5793 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Practitioner$PractitionerBuilder.class
  1470 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/CodeableConcept.class
  1442 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Element.class
  3136 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QUriType.class
   884 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/PractitionerHistory.class
  2739 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/DomainResource$DomainResourceBuilder.class
   438 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/IntegerType.class
  5421 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QPractitionerRole.class
  5842 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Organization.class
  3662 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/ContactPoint.class
   429 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/DateType.class
   471 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Enumerations.class
  6853 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Practitioner.class
  9473 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Patient.class
  4334 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QHumanName.class
  4152 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QContactPoint.class
  2913 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QType.class
  2321 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Practitioner$PatientReference$PatientReferenceBuilder.class
  2125 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Patient$GeneralPractitionerReference.class
  3268 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QDateType.class
  6033 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QPatientHistory.class
  3299 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QPrimitiveType.class
  1352 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/IdType.class
  5694 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/SearchParameter.class
  4037 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QCoding.class
  2855 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QPatient_GeneralPractitionerReference.class
  5256 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QPatient_ContactComponent.class
  2794 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Resource$ResourceBuilder.class
   978 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/PrimitiveType.class
  3404 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QResource.class
  2786 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QPractitionerRole_PractitionerReference.class
  1535 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/HumanName$NameUse.class
  3179 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QBooleanType.class
   884 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/OrganizationHistory.class
  4606 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/PractitionerRole.class
  2474 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/Organization$ContactComponent.class
  3308 Mon Mar 07 12:28:00 CET 2022 me/jeusdi/fhir/hazelcast/model/QDateTimeType.class
  8042 Mon Mar 07 12:27:46 CET 2022 META-INF/maven/me.jeusdi.fhir.hazelcast/fhir-hazelcast/pom.xml
    82 Mon Mar 07 12:28:02 CET 2022 META-INF/maven/me.jeusdi.fhir.hazelcast/fhir-hazelcast/pom.properties

EDIT 2: I've also tried to use console provided by mancenter:

As you can see, I'm able to get my object stored into practitioner map.

However, when I navigate to map browser:

I'm getting: