Sending protobuf as JSON in spring-boot

Question

I´m using protobufs with this concrete definition.

message Hash {
    string category = 1;
    repeated KVPair content = 2;
}

message KVPair {
    string key = 1;
    string value = 2;
}

I want to send this as JSON with my spring-boot application. I added this package to my gradle dependencies:

compile group: 'com.google.protobuf', name: 'protobuf-java', version: '3.6.1'

When i try to output Hash generated object with this code:

@RestController
@RequestMapping("/api/crm/")
public class KVController {

    private final KVService kvService;

    public KVController(KVService kvService) {
        this.kvService = kvService;
    }

    @GetMapping("kv/{category}")
    public Hash getHash(@PathVariable String category) {
        Hash hash = kvService.retrieve(category);
        return hash;
    }
}

It throws this ultimate exception:

Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Direct self-reference leading to cycle (through reference chain: com.blaazha.crm.proto.Hash["unknownFields"]->com.google.protobuf.UnknownFieldSet["defaultInstanceForType"]) at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1191) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.BeanPropertyWriter._handleSelfReference(BeanPropertyWriter.java:944) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:721) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396) ~[jackson-databind-2.9.6.jar:2.9.6] at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:913) ~[jackson-databind-2.9.6.jar:2.9.6] at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:286) ~[spring-web-4.3.18.RELEASE.jar:4.3.18.RELEASE] ... 58 common frames omitted

kvService only returns data from redis. It parses Hash data type (https://redis.io/topics/data-types) to Hash object defined in proto. Where Hash->category is main key of hash and values in hash redis datatype are converted to KVPair defined in proto. I cannot show all source code, because it calls other systems and source code is very long.

kvService returns valid Hash object, but exception happens when I return this Hash object and spring tries convert it to JSON.

important dependencies in my build.gradle:

def versions = [
        logback: '1.2.3',
        owner: '1.0.10',
        jackson: '2.9.6',

        guava: '25.1-jre',
        guice: '4.2.0',
        grpc: '1.9.1',
        protoc: '3.5.1',

        redis: '2.9.0',
]

dependencies {

compile group: 'ch.qos.logback', name: 'logback-classic', version: versions.logback
compile group: 'org.aeonbits.owner', name: 'owner', version: versions.owner

compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: versions.jackson
compile group: 'com.fasterxml.jackson.core', name: 'jackson-annotations', version: versions.jackson
compile group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-yaml', version: versions.jackson

compile group: 'com.google.guava', name: 'guava', version: versions.guava
compile group: 'com.google.inject', name: 'guice', version: versions.guice
compile group: 'io.grpc', name: 'grpc-netty', version: versions.grpc
compile group: 'io.grpc', name: 'grpc-protobuf', version: versions.grpc
compile group: 'io.grpc', name: 'grpc-stub', version: versions.grpc
compile 'org.glassfish:javax.annotation:10.0-b28'


compile group: 'javax.xml.bind', name: 'jaxb-api', version: '2.2.1'
compile group: 'javax.activation', name: 'activation', version: '1.1.1'

compile group: 'redis.clients', name: 'jedis', version: versions.redis

}

As you can see in my protobuf definition isn´t any self-referencing.

Is there any possible way to fix this problem ?

In which line of code happens the exception? Could you please share your kvservice code? I think it will be easier for people to answer your question with that information in place. — f-CJ, Aug 27 '18 at 18:09

score 7 · Answer 1 · answered Apr 28 '20 at 20:45

If you're using Spring WebFlux and trying to produces application/json here is what you can do to make it works for all mappings returning protobuf Message type:

@Configuration
@EnableWebFlux
public class WebConfig implements WebFluxConfigurer {

@Override
public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) {
    configurer.defaultCodecs().jackson2JsonEncoder(
        new Jackson2JsonEncoder(Jackson2ObjectMapperBuilder.json().serializerByType(
                Message.class, new JsonSerializer<Message>() {
                    @Override
                    public void serialize(Message value, JsonGenerator gen, SerializerProvider serializers) throws IOException {
                        String str = JsonFormat.printer().omittingInsignificantWhitespace().print(value);
                        gen.writeRawValue(str);
                    }
                }
        ).build())
    );
}

It works perfectly for serialization, any idea how to accommodate for deserialization. — Karthik Prasad, Jun 06 '20 at 15:39

vlp · Accepted Answer · 2019-05-09T23:06:10.887

4

Class UnknownFieldSet (reached via generated method Hash.getUnknownFields()) contains getter getDefaultInstanceForType() which returns singleton instance of UnknownFieldSet. This singleton instance references itself in getDefaultInstanceForType() and Jackson-databind can't handle this automatically (see edit2 below).

You might want to use JsonFormat from com.google.protobuf:protobuf-java-util which uses canonical encoding instead of Jackson.

Good luck!

EDIT> For Spring there is ProtobufJsonFormatHttpMessageConverter

EDIT2> Of course you could handle this situation using Mix-in Annotations, but IMHO JsonFormat is definitely the way to go...

edited May 09 '19 at 23:06

answered Aug 27 '18 at 23:22

vlp

7,811
2
23
51

Is that worked? Where should I update httpmessageconverter? – Prasath Feb 08 '19 at 09:31
1

@Prasath Have a look e.g. [here](http://www.jcombat.com/spring/understanding-http-message-converters-in-spring-framework) or [here](https://www.baeldung.com/spring-httpmessageconverter-rest). Good luck! – vlp Feb 10 '19 at 10:25

score 2 · Answer 3 · answered Jul 21 '20 at 16:21

2

To convert the protobuf object to JSON, you should be using the following class from the package com.google.protobuf.util.JsonFormat as:

JsonFormat.printer().print()

answered Jul 21 '20 at 16:21

Avinash Reddy Penugonda

159
2
12

score 1 · Answer 4 · answered Apr 22 '22 at 11:35

I was able to resolve my issue by adding a bean like so in my Spring main application.

Tried a number of answers on StackOverflow with no luck. You may have some success by adding the protobuf types to the registry as defined here, as some of other common answers did not work for me.

@Bean
public Jackson2ObjectMapperBuilderCustomizer jackson2ObjectMapperBuilderCustomizer() {
    JsonFormat.TypeRegistry typeRegistry = JsonFormat.TypeRegistry.newBuilder()
            .add(MyType.getDescriptor())
            .add(MyOtherType.getDescriptor())
            .build();

    JsonFormat.Printer printer = JsonFormat.printer().usingTypeRegistry(typeRegistry)
            .includingDefaultValueFields();
    return o -> o.serializerByType(Message.class, new JsonSerializer<Message>() {
        @Override
        public void serialize(Message message, JsonGenerator gen, SerializerProvider serializers) throws IOException {
            gen.writeRawValue(printer.print(message));
        }
    });
}

score 1 · Answer 5 · answered Nov 02 '22 at 11:18

An example based on vlp's answer. Do the following:

Declare a Bean of type ProtobufJsonFormatHttpMessageConverter. e.g.

@Configuration
public class AppConfig {
    @Bean
    public ProtobufJsonFormatHttpMessageConverter protobufHttpMessageConverter() {
        return new ProtobufJsonFormatHttpMessageConverter(
                JsonFormat.parser().ignoringUnknownFields(),
                JsonFormat.printer().omittingInsignificantWhitespace()
        );
    }
}

Add produces = MediaType.APPLICATION_JSON_VALUE to your controller method. e.g.

@GetMapping(value = "/get-one-item", produces = MediaType.APPLICATION_JSON_VALUE)
public Item getItems() {
    //Item is a Protobuf message
    return itemService.getOneItem();
}

score 0 · Answer 6 · answered Aug 07 '23 at 16:10

With Spring Boot Web MVC you need to create a converter

@Component
public class ProtobufToJsonConverter extends AbstractHttpMessageConverter<Message> {

  private final JsonFormat.Printer printer;

  private final JsonFormat.Parser parser;

  public ProtobufToJsonConverter() {
    parser = JsonFormat.parser()
      .ignoringUnknownFields();
    printer = JsonFormat.printer()
      .includingDefaultValueFields()
      .omittingInsignificantWhitespace();
  }

  @Override
  protected boolean canRead(MediaType mediaType) {

    return MediaType.APPLICATION_JSON.equals(mediaType);
  }

  @Override
  protected boolean canWrite(MediaType mediaType) {

    return MediaType.APPLICATION_JSON.equals(mediaType);
  }

  @Override
  protected @NotNull Message readInternal(@NotNull Class<? extends Message> clazz, @NotNull HttpInputMessage inputMessage) throws IOException, HttpMessageNotReadableException {

    try {
      final Message.Builder builder = (Message.Builder)clazz.getDeclaredMethod("newBuilder").invoke(null);
      parser
        .merge(new InputStreamReader(inputMessage.getBody()), builder);
      return builder.build();
    } catch (ReflectiveOperationException e) {
      throw new IOException(e);
    }
  }

  @Override
  protected boolean supports(@NotNull Class<?> clazz) {

    return Message.class.isAssignableFrom(clazz);
  }

  @Override
  protected void writeInternal(@NotNull Message message, @NotNull HttpOutputMessage outputMessage) throws IOException, HttpMessageNotWritableException {

    outputMessage.getBody().write(printer.print(message).getBytes(StandardCharsets.UTF_8));

  }


}

There's no need to register it but the component must be scanned by having it in the same or child package as your @SpringBootApplication or @ComponentScan. Once it is scanned it is available for any HTTP conversions.

Sending protobuf as JSON in spring-boot

6 Answers6