2

I am trying to integrate spring cloud stream with spring cloud function webflux

as they are deprecating spring cloud reactive streams in future releases I am trying to use spring cloud functions https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/2.1.2.RELEASE/single/spring-cloud-stream.html#spring-cloud-stream-preface-notable-deprecations

Spring cloud web function can expose an end point of its function with paths like in the doc

https://cloud.spring.io/spring-cloud-static/spring-cloud-function/1.0.0.RELEASE/single/spring-cloud-function.html

from cloud stream I can see the source needs to be defined as supplier https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/2.1.2.RELEASE/single/spring-cloud-stream.html#_spring_cloud_function

but my use case is to get POST data from reactive http end point and ingest into kafka, is there any way achieving it from spring cloud function web and spring cloud stream ?

from the doc for spring cloud function with spring cloud stream

@SpringBootApplication
@EnableBinding(Source.class)
public static class SourceFromSupplier {
    public static void main(String[] args) {
        SpringApplication.run(SourceFromSupplier.class, "--spring.cloud.stream.function.definition=date");
    }
    @Bean
    public Supplier<Date> date() {
        return () -> new Date(12345L);
    }
}

and if i run this i can see date is getting inserted into kafka every 1 second and if i call the get endpoint for supplier like localhost:/8080/date results in a date response, is there any way of injecting the paylaod from post to kafka with spring cloud function ?

jay
  • 73
  • 8

1 Answers1

1

There is an issue which your question helped to discover and it has to do with lifecycle inconsistency between auto-configurations provided by function and stream. The issue manifests itself in a way that the rest point created by Spring Cloud Functions can not see the bindings as it is created much earlier

So we'll address the issue shortly. Meanwhile there is a workaround which would require you to access output channel from the ApplicationContext (see below):

@SpringBootApplication
@EnableBinding(Source.class)
public class SimpleFunctionRabbitDemoApplication {

  public static void main(String[] args) throws Exception {      
    SpringApplication.run(SimpleFunctionRabbitDemoApplication.class);
  }

  @Bean
  public Consumer<String> storeSync(ApplicationContext context) {
     return v -> {
        MessageChannel channel = context.getBean(Source.OUTPUT, MessageChannel.class);
        channel.send(MessageBuilder.withPayload(v).build());
     };
  }
}
Oleg Zhurakousky
  • 5,820
  • 16
  • 17
  • Thanks for the quick reply but still i am not able to inject in kafka streams, i have exposed a bean of type Consumer for the spring cloud function and when i post some data to it I can see the data in the function but when i call the sink.input().send(MessageBuilder.withPayload(paylaod).build()) i was not able to send the message to kafka but i have received true when i log it. i am using spring boot 2.1.3-release,spring cloud stream Fishtown.SR2, springCloudFunctionVersion 2.0.1.RELEASE – jay Mar 27 '19 at 19:30
  • Consider creating a small sample which demonstrates your issue and push i to GitHub, this way we can take a look – Oleg Zhurakousky Mar 27 '19 at 19:46
  • hi please find my github sample[ https://github.com/jayasai470/spring-sample-cloud-stream-function] – jay Mar 27 '19 at 20:36
  • try this please https://github.com/jayasai470/spring-sample-cloud-stream-function – jay Mar 27 '19 at 20:48
  • Ok, i'll take a look tomorrow, meanwhile, please upgrade to the latests Stream which was just released as 2.2.0.M1 - https://spring.io/blog/2019/03/26/spring-cloud-stream-germantown-m1-2-2-0-m1-release-announcement – Oleg Zhurakousky Mar 27 '19 at 20:53
  • Any suggestions? – jay Mar 29 '19 at 13:31
  • Yes, I've just updated the answer and raise the issue in stream - https://github.com/spring-cloud/spring-cloud-stream/issues/1678 – Oleg Zhurakousky Mar 29 '19 at 15:03
  • Just tried your suggestions but no success, i have pushed the changes to my repo, its still behaving the same like when i trigger my rest api its stuck in an infinite loop from the logs i can see the channel.send returned a true even though i did not see any messages in kafka – jay Mar 29 '19 at 15:58
  • i just had to remove my --spring.cloud.stream.function.definition=storeSync and it worked, thanks for the suggestion – jay Mar 29 '19 at 16:08
  • channel.send is a blocking call, is there any reactive api to inject into kafka stream, i can only find this from doc https://cloud.spring.io/spring-cloud-static/spring-cloud-stream/2.1.2.RELEASE/single/spring-cloud-stream.html#_reactive_sources – jay Mar 29 '19 at 16:31
  • I'd suggest rasing an issue in [spring-cloud-stream](https://github.com/spring-cloud/spring-cloud-stream/issues). I thought about it as well, but it seems like something we'd need to implement – Oleg Zhurakousky Mar 29 '19 at 17:47
  • ok just opened it https://github.com/spring-cloud/spring-cloud-stream/issues/1679 – jay Mar 29 '19 at 19:27
  • insertion errors in kafka binder level are not thrown in the cloud function webflux, is there any way we can get exception in channel.send method ? – jay Mar 30 '19 at 05:36