I've mostly successfully set up a test using EmbeddedKafka, which produces and consumes one message. Below is a working version of the test:
@RunWith(SpringRunner.class)
@SpringBootTest
@EmbeddedKafka
public class KafkaFlowTest {
@Autowired
EmbeddedKafkaBroker broker;
public static final String EMAIL_TOPIC = "email-service";
static {
System.setProperty(EmbeddedKafkaBroker.BROKER_LIST_PROPERTY, "spring.kafka.bootstrap-servers");
}
private Consumer<String, KafkaEmailMessageWrapper> consumer;
private Producer<String, KafkaEmailMessageWrapper> producer;
@Before
public void setUp() {
Map<String, Object> consumerProps = KafkaTestUtils.consumerProps("group", "true", broker);
consumerProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
consumerProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
consumerProps.put(JsonDeserializer.TRUSTED_PACKAGES, "*");
Map<String, Object> producerProps = KafkaTestUtils.producerProps(broker);
producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
DefaultKafkaConsumerFactory<String, KafkaEmailMessageWrapper> cf =
new DefaultKafkaConsumerFactory<>(
consumerProps,
new StringDeserializer(),
new JsonDeserializer<>(KafkaEmailMessageWrapper.class)
);
consumer = cf.createConsumer();
consumer.subscribe(Collections.singleton(EMAIL_TOPIC));
consumer.poll(Duration.ZERO);
DefaultKafkaProducerFactory<String, KafkaEmailMessageWrapper> pf =
new DefaultKafkaProducerFactory<>(producerProps);
producer = pf.createProducer();
}
@Test
public void testNoErrorMessageFlow() {
KafkaEmailMessageWrapper wrapperMessage = KafkaEmailMessageWrapper.builder()
.emailRequest(
EmailMessage.builder()
.body("body")
.from("me@me.com")
.to(new String[]{"you@you.com"})
.subject("hey!")
.build()
)
.build();
producer.send(new ProducerRecord<>(EMAIL_TOPIC, "whatever", wrapperMessage));
producer.flush();
ConsumerRecord<String, KafkaEmailMessageWrapper> record = KafkaTestUtils.getSingleRecord(consumer, EMAIL_TOPIC);
KafkaEmailMessageWrapper receivedMessage = record.value();
assertEquals(wrapperMessage.getEmailRequest().getSubject(), receivedMessage.getEmailRequest().getSubject());
}
}
This works fine, unless I add assertEquals on the to field, which in JSON comes as a list of email addresses separated by ; and in the class is an array of Strings. So in the EmailMessage class I have the to field annotated with @JsonDeserializer pointing to a custom deserializer that splits along ;, etc. Everything works when the app is run, only tests break.
I tried to modify the code above to change the Producer<String, KafkaEmailMessageWrapper> producer to private Producer<String, String> producer and then in the test send JSON instead of the KafkaEmailMessageWrapper instance, but then I get an exception:
java.lang.ClassCastException: java.lang.String cannot be cast to com.test.model.KafkaEmailMessageWrapper
So it looks like the deserialization from JSON string to the model is for some reason not happening in this test scenario. I would like the test be as close to real use case as possible, so it should produce a string message and then perform deserialization to my model class. Not sure why this is not happening, any help in understanding why that is would be appreciated!
For completion, this is the definition of message listener:
@KafkaListener(topics = "email-service")
public void receive(@Payload KafkaEmailMessageWrapper message)
EDIT
Let me rephrase the problem, because maybe I confused things a bit. Overview of the app is:
- consume a JSON message
- serialize it into
KafkaEmailMessageWrapper - send an email
This works, I send a JSON to Kafka, it's received (listener definition above) and proceeds correctly. The only problem I have is while testing.
I want to mimic the same situation in the test, but when I do this:
kafkaTemplate.sendDefault("key", "<json message>");
I get a:
Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [java.lang.String] to [com.test.model.KafkaEmailMessageWrapper] for GenericMessage
However when I send a KafkaEmailMessageWrapper instance instead:
kafkaTemplate.sendDefault("key", kafkaEmailMessageWrapper);
it works, but then it doesn't use my @JsonDeserialize annotations I have specified on the class:
@JsonDeserialize(using = SemicolonArrayDeserializer.class)
private String[] to;
@JsonDeserialize(using = SemicolonArrayDeserializer.class)
private String[] cc;
So when sending an instance, to and cc fields are always empty, because in the fields already are arrays of strings, so the deserialize logic of splitting along ; is not doing anything. This is the deserialize method:
public String[] deserialize(JsonParser p, DeserializationContext ctxt) throws IOException {
ObjectMapper mapper = (ObjectMapper) p.getCodec();
JsonNode node = mapper.readTree(p);
return node.asText().split(";");
}
