Stack Overflow
Stack Overflow

Questions tagged [spring-cloud-dataflow]

Spring Cloud Data Flow is a toolkit for building data integration and real-time data processing pipelines. Pipelines consist of Spring Boot apps, built using the Spring Cloud Stream or Spring Cloud Task microservice frameworks. This makes Spring Cloud Data Flow suitable for a range of data processing use cases, from import/export to event streaming and predictive analytics.

Use this tag for questions about the Spring Cloud Data Flow project. It is not intended for general questions about integrating other Spring projects with other technologies.

Spring Cloud Data Flow's Official Project Site

Spring Cloud Data Flow Website

Spring Cloud Data Flow's Github Repo

How to contribute

How to report issues

Related tags , , , , , .

1276 questions
2
votes
1 answer

Schedule simple GET batch for each second or even less than one second - Should opt for Spring Cloud Task, Spring Batch or springframework.scheduling

Context: in my country there will be a new way to Instantly Payment previewed for November. Basically, the Central Bank will provide two endpoints: (1) one POST endpoint which we post a single money transfer and (2) one GET endpoint where we get the…
Jim C
  • 3,957
  • 25
  • 85
  • 162
2
votes
2 answers

DB Credentials Exposed as part job parameters when executing a task from SCDF

I have Custom Built SCDF which is built as docker image in Openshift and referred in server-deployment.yaml as docker image.I use the Oracle db to store the task meta data and is an external source here. I pass the all db properties in configmap.…
Rajesh2389
  • 349
  • 2
  • 15
2
votes
1 answer

org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:7577/api/release/states":

I'm working on Spring Batch and Spring Cloud Data Flow example. In this example, I've created a spring batch project having single job for now it reads data from flatfile and load data into mongodb . I was successfully able to trigger the batch job…
Jeff Cook
  • 7,956
  • 36
  • 115
  • 186
2
votes
1 answer

How to add Oracle Driver dependency to Spring Data Flow Server?

I'm currently trying Spring Data Flow Server with Spring Batches. I have two datasources in my application.properties. And the Spring Batch application works fine and it reads and writes data into the Database. But When I tried to deploy the Jar…
Rajesh2389
  • 349
  • 2
  • 15
2
votes
1 answer

A dynamic message orchestration flow engine for Kafka

I'm trying to see what are the possible toolkits/frameworks available to achieve the following. A toolkit where a developer typically should configure the data flow (which is a series of steps) to form a data processing pipeline. A declarative…
user1189332
  • 1,773
  • 4
  • 26
  • 46
2
votes
1 answer

How to keep log file in Spring Cloud Data Flow?

My Spring Cloud Data Flow deleted log file in folder after I stopped it. Why SCDF does that and How can I keep these log files?
Zach Pham
  • 280
  • 4
  • 18
2
votes
2 answers

Spring cloud data flow custom application properties

I created a custom spring cloud data flow application. I would like to create a stream with it and put some application properties in it, as we can add for the provided application log (0/3 properties): I tried with application.yml file in…
marie
  • 457
  • 8
  • 27
2
votes
1 answer

Spring cloud dataflow use H2 database

I try to use postgresql with Spring cloud dataflow using this connection: java -jar spring-cloud-dataflow-server-2.2.2.BUILD-SNAPSHOT.jar --spring.datasource.url=jdbc:postgresql://localhost:5432/test --spring.datasource.username=postgres…
LyLy
  • 41
  • 2
2
votes
0 answers

Change service name of Spring Cloud Dataflow Stream parts

When I deploy a new stream in Spring Cloud Dataflow I want to influence the service name, so that I can reach e.g. the source always under the same name. I already found this this anwer but it doesn't mention how to turn this off. Looking at the…
Ronny
  • 31
  • 4
2
votes
0 answers

Can not run "egrep" while building maven project

I want to run the project to schedule task on Spring Cloud Data Flow local I met this error while building maven project: [ERROR] Command execution failed. java.io.IOException: Cannot run program "egrep" (in directory…
Zach Pham
  • 280
  • 4
  • 18
2
votes
2 answers

Spring data flow skipper failed to start after deployment to CF

I have an issue that after pushing the applications, both scdf server and skipper, fails to start due to authentication error. This is related to the internal scdf user (deployer user) which stop/start that tasks/apps. The user that set in the yml…
ShalomYa
  • 41
  • 6
2
votes
2 answers

Different retry strategies for different consumers in Kafka

We are dealing with a scenario where we need different retry strategies for different consumers in same application. Please refer to below figure (a brief architecture diagram): The main_consumer consumes payload from main_topic and tries to send…
Akhil Ghatiki
  • 1,140
  • 12
  • 29
2
votes
2 answers

Failed to start bean 'taskLifecycleListener'; nested exception is java.lang.IllegalArgumentException: Invalid TaskExecution, ID 1 not found

Problem : I am working with spring boot project and make as @Enabletask and tried to run with spring data flow. I have up the spring dataflow server and in shell i tried to registered app and created task and when i run task i got the error. By…
Path Mehta
  • 21
  • 3
2
votes
3 answers

Error with docker-compose up for Spring Cloud Dataflow - "DATAFLOW_VERSION is not set!"

I am a newbie to spring-cloud-data-flow. I am following the documentation of Spring Cloud Data Flow (https://dataflow.spring.io/docs/installation/local/docker/). I have downloaded docker-compose.yml file and put it in D:\Dev\spring-cloud-dataflow>…
selins sofa
  • 76
  • 7
2
votes
2 answers

Can't change content_type from application/octet-stream to application/json in RabbitMQ sink

I have a simple RabbitMQ source and sink. I'm publishing a message to the source queue with the following properties: content_type -> application/json and a JSON payload: { "userId": 2, "customerId": 1, } The RabbitMQ sink gets the message…
Maroun
  • 94,125
  • 30
  • 188
  • 241
1 2 3
…
84 85