12

I am trying to implement a simple Kafka communication between 2 different Spring Boot applications with out any special settings, this application has only one kafkalistener. My yml for the consumer is the following:

spring:
  kafka:
    bootstrap-servers: ip_here
    topic:
      json: topic_here
    consumer:
      group-id: group_id
      auto-offset-reset: earliest
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring:
          json:
            trusted:
              packages: 'com.example.kw.dtos.Classdata'

The error I am receiving is the following:

Caused by: java.lang.IllegalArgumentException: The class 'com.example.kw.dtos.Classdata' is not in the trusted packages: [java.util, java.lang, com.example.kw.dtos.Classdata]. If you believe this class is safe to deserialize, please provide its name. If the serialization is only done by a trusted source, you can also enable trust all (*).

The package is in the trusted packages but something is wrong.

My factory class:

@Configuration
@EnableKafka
public class MsgListener {

    @Value("${spring.kafka.bootstrap-servers}")
    private String bootstrapServers;

    @Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "json");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        props.put(JsonDeserializer.TRUSTED_PACKAGES, "com.example.kw.dtos.Classdata");
        return props;
    }

    @Bean
    public ConsumerFactory<String, Classdata> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(
                consumerConfigs(),
                new StringDeserializer(),
                new JsonDeserializer<>(Classdata.class));
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, Classdata> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, Classdata> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }
}
HeyItsMe
  • 191
  • 1
  • 1
  • 10
  • where are u using the trusted packages property? – Indraneel Bende May 27 '18 at 15:06
  • I am not using it in any place. – HeyItsMe May 27 '18 at 15:24
  • Is it possible for you to add your code where your creating kafka consumer factory? – Indraneel Bende May 27 '18 at 15:26
  • I just added a factory class. It was missing. Should the trusted packages exist in the producer as well??? – HeyItsMe May 27 '18 at 15:42
  • are u creating a json representation of a java object and then producing, and then when consuming converting from json string back into a java object? – Indraneel Bende May 27 '18 at 15:48
  • https://kafka.apache.org/10/javadoc/index.html?org/apache/kafka/common/serialization/Deserializer.html I dont see JsonDeserializier here. – Indraneel Bende May 27 '18 at 15:48
  • The constructor of DefaultKafkaConsumerFactory expects the 3rd argument in the constructor to be a implementation of the above interface. That , is why it is failing. – Indraneel Bende May 27 '18 at 15:52
  • This is exactly what I am doing. I receive a POST request mapping it into an object using @RequestBody and immediately send to kafka topic. Then I am expecting to receive it from another application and this is where is breaks. – HeyItsMe May 27 '18 at 15:54
  • okay now i know, you are using this https://docs.spring.io/spring-kafka/api/org/springframework/kafka/support/serializer/JsonDeserializer.html Have u got a JsonSerializer specified in the ProducerFactory? – Indraneel Bende May 27 '18 at 15:58
  • No I haven't specified a factory in the producer application. Only simple kafka template. – HeyItsMe May 27 '18 at 16:00
  • Create a KafkaProducerFactory yourself and as the value serializer specify JsonSerializer. – Indraneel Bende May 27 '18 at 16:01

3 Answers3

7

It should be just the package com.example.kw.dtos

String packageName = ClassUtils.getPackageName(requestedType).replaceFirst("\\[L", "");
for (String trustedPackage : this.trustedPackages) {
    if (packageName.equals(trustedPackage)) {
        return true;
    }
}
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • That did work but now because I am using a different package name there is a MessageConversionException: failed to resolve class name. – HeyItsMe May 27 '18 at 16:23
  • 2
    When using a `@KafkaListener` with a different object to that sent by the producer, you should use a `StringJsonMessageConverter` with a `StringDeserializer` or `ByteArrayDeserializer` instead of a `JsonDeserializer` - see [this answer](https://stackoverflow.com/questions/50478267/classnotfoundexception-with-kafka-consumer/50489352#50489352). The actual parameter type is passed into the converter. – Gary Russell May 27 '18 at 16:45
2

We had this issue while testing kafka. We fixed it that way:

private static KafkaMessageListenerContainer<String, Data> createMessageListenerContainer() {
  final Map<String, Object> consumerProps = KafkaTestUtils.consumerProps("sender", "false", EMBEDDED_KAFKA);
  final DefaultKafkaConsumerFactory<String, Data> consumerFactory = new DefaultKafkaConsumerFactory<>(consumerProps);

  final JsonDeserializer<Data> valueDeserializer = new JsonDeserializer<>();
  valueDeserializer.addTrustedPackages("path.to.package");

  consumerFactory.setValueDeserializer(valueDeserializer);
  consumerFactory.setKeyDeserializer(new StringDeserializer());

  final ContainerProperties containerProperties = new ContainerProperties(SENDER_TOPIC);
  return new KafkaMessageListenerContainer<>(consumerFactory, containerProperties);
}
Haagy
  • 61
  • 1
  • 9
0

The trick here is that you have to set it in two places

  • spring.json.trusted.packages - for any json deserializers created outside of kafka's influence
  • spring.kafka.consumer.properties.spring.json.trusted.packages - for kafka created deserializers

This was the only way i was able to make it work. Also, it does not accept wildcards, so it has to be exact package match

Ján Srniček
  • 505
  • 1
  • 10
  • 34
  • The first configuration yields `The configuration 'spring.json.trusted.packages' was supplied but isn't a known config. - {} `. It does not seem to be required. – ojathelonius Feb 21 '22 at 17:17