Stack Overflow
Stack Overflow

Questions tagged [spring-cloud-dataflow]

Spring Cloud Data Flow is a toolkit for building data integration and real-time data processing pipelines. Pipelines consist of Spring Boot apps, built using the Spring Cloud Stream or Spring Cloud Task microservice frameworks. This makes Spring Cloud Data Flow suitable for a range of data processing use cases, from import/export to event streaming and predictive analytics.

Use this tag for questions about the Spring Cloud Data Flow project. It is not intended for general questions about integrating other Spring projects with other technologies.

Spring Cloud Data Flow's Official Project Site

Spring Cloud Data Flow Website

Spring Cloud Data Flow's Github Repo

How to contribute

How to report issues

Related tags , , , , , .

1276 questions
0
votes
1 answer

Spring Cloud Dataflow - Setting Stream Apps Env Variables

I'm deploying a Spring Cloud Dataflow stream onto Cloud Foundry. I need to set an environment variable for the underlying stream applications. I know that putting in configuration into the stream definition will update the SPRING_APPLICATION_JSON…
user6448130
0
votes
1 answer

spring-cloud-starter-stream-source-file delete the file after processing

I'm trying to utilize spring-cloud-starter-stream-source-file app (https://github.com/spring-cloud-stream-app-starters/file/tree/master/spring-cloud-starter-stream-source-file) to poll a directory and send the file ref to custom processor for…
Pons
  • 1,101
  • 1
  • 11
  • 20
0
votes
1 answer

Spring Cloud Dataflow Splitter with JSON Array

I'm trying to use the Splitter app to split a JSON array e.g. [{...},{...}] into multiple messages {...} {...}. With input contentType=application/json (per the docs) Spring Cloud is surfacing an exception from Jackson:…
Kevin Niemann
  • 51
  • 8
0
votes
1 answer

Cannot deploy custom module that uses the same property name, but under different prefixes

System Information Spring Cloud Data Flow Cloud Foundry: v1.1.0.RELEASE Pivotal Cloud Foundry: v1.7.12 CF Client (Windows): cf.exe version 6.23.1+a70deb3.2017-01-13 cf-v3-plugin: 0.6.7 I have developed a custom source module that connects to a REST…
Paul Barriscale
  • 21
  • 6
0
votes
1 answer

Programmatically getting the stream name

System Information Spring Cloud Data Flow Cloud Foundry: v1.1.0.RELEASE Pivotal Cloud Foundry: v1.7.12 CF Client (Windows): cf.exe version 6.23.1+a70deb3.2017-01-13 cf-v3-plugin: 0.6.7 I would like to inject the stream name into a bean defined in my…
Paul Barriscale
  • 21
  • 6
0
votes
0 answers

Deploy spring-cloud-dataflow stream oracle2hdfs not well (ambari)

Thanks advance for your help. I deploy spring-cloud-dataflow-server-yarn with ambari. But when I start a stream jdbc-source-kafka and hdfs-sink-kafka. The stream deploy not run. The streams config jdbc-source-kafka --max-rows-per-poll=10…
user3172755
  • 137
  • 1
  • 10
0
votes
0 answers

Can't deploy stream or task in Spring Cloud Data Flow successfully

I have a spark cluster in yarn mode, and I installed Spring Cloud Data Flow on the master node directly, followed the simple steps of the document, everything goes well. I created simple stream and task, deployed them, nothing happened…
Lionel
  • 13
  • 2
0
votes
0 answers

kubernetes-scdf error: [io-9393-exec-10] o.s.c.d.s.k.AbstractKubernetesDeployer : An error has occurred

I created a spring-cloud-dataflow pod on my own k8s cluster . And the error occured when I deploy a stream on the scdf. Logs are the below : 2017-02-01 15:04:56.121 ERROR 1 --- [nio-9393-exec-5] o.s.c.d.s.k.AbstractKubernetesDeployer : An error…
Oion Power
  • 41
  • 1
  • 5
0
votes
1 answer

Spring Cloud Dataflow with multiple Kafka binders

I'm trying to use Spring Cloud Dataflow to bridge two Kafka clusters (essentially a fancy MirrorMaker instance) using the Bridge app. As covered in the docs, I've defined two binders. Kafka-qa1 should be the default, and kafka-qa2 can be provided in…
Kevin Niemann
  • 51
  • 8
0
votes
0 answers

Getting ReplyRequiredException in Spring cloud dataflow

I am trying out spring cloud dataflow. I created a stream with 'source' as "maven://org.springframework.cloud.stream.app:file-source-kafka:1.0.2.RELEASE", I have written my custom processor and the 'sink' is…
krajwade
  • 133
  • 1
  • 12
0
votes
1 answer

Spring Cloud Data Flow programmable application's error channel

How can I define or program an app's error channel that will receive all messages that have failed in processors/sinks? The documentation says the following: When the number of retry attempts has exceeded the maxAttempts value, the exception and…
Yuriy Tkach
  • 78
  • 1
  • 7
0
votes
1 answer

Spring DataFlow Yarn - Container is running beyond physical memory

I'm running Spring Cloud Tasks on Yarn simple tasks work fine but running bigger tasks which require more resources I got "Container is running beyond physical memory" error: onContainerCompleted:ContainerStatus: [ContainerId:…
Ragnar
  • 55
  • 1
  • 8
0
votes
1 answer

Spring Cloud Dataflow send buffered messages to output channel

I have a filter which buffers rejected messages. I would like to send these buffered messages to the output channel once my predicate is satisfied: public class MyFilter implements MessageSelector { private Buffer buff; ... @Override …
alturkovic
  • 990
  • 8
  • 31
0
votes
1 answer

Redis datasource for spring cloud dataflow server

As per the documentation, Spring Cloud Dataflow Service used RDBMS for storing stream/task definitions, application registration and job repositories. Instead of using RDBMS, is there a way to use Redis for storing this information.
krajwade
  • 133
  • 1
  • 12
0
votes
1 answer

Aggregate-counter on an existing stream

I'm trying to create an aggregate counter for various streams I have set up. In SpringXD it would look like this: "tap:stream:MyCustomStream > aggregate-counter". In Spring Cloud Dataflow so far I have done ":MyKafkaTopic > aggregate-counter", which…
Kevin Niemann
  • 51
  • 8
1 2 3
…
84 85