Stack Overflow
Stack Overflow

Questions tagged [spring-xd]

Spring XD is a unified, distributed, and extensible system for data ingestion, real time analytics, batch processing, and data export. The project’s goal is to simplify the development of big data applications.

Spring XD is a unified, distributed, and extensible system for data ingestion, real time analytics, batch processing, and data export. The project’s goal is to simplify the development of big data applications.

Big data applications share many characteristics with Enterprise Integration and Batch applications. Spring has provided proven solutions for building integration and batch applications for more than 6 years now via the Spring Integration and Spring Batch projects. Spring XD builds upon this foundation and provides a lightweight runtime environment that is easily configured and assembled via a simple DSL.

Click here for more information about Spring XD.

565 questions
1
vote
1 answer

Headers in http source Spring cloud flow

I am using postman to POST a http request with a file in form-data and want to pass the filename as a header. With file as a sink i want to get that file and save it in the name of the header value filename which i passed in the http request. Below…
Vignesh I
  • 2,211
  • 2
  • 20
  • 40
1
vote
1 answer

springxd stream using HDFS-Dataset to save avro data unable to renew kerberos ticket

I have created a springxd stream ====> source-JMS queue -> Transform-Custom Java Processor (XML to AVRO) -> Sink - HDFS-Dataset. Stream works perfectly fine but after 24 hours, since its continuous connection it is unable to renew the kerberos…
AJ007
  • 125
  • 2
  • 11
1
vote
1 answer

Spring Integration & Kafka Consumer: Stop message-driven-channel-adapter right after records are sucessfully fetched

I'm using the following confing: spring-integration-kafka 2.1.0.RELEASE kafka-clients 0.10.0.1 Kafka 0.10.x.x spring-xd-1.3.1.RELEASE I created my custom Kafka-source module for SpringXD. I set my consumer logic and my…
Columb1a
  • 463
  • 2
  • 11
  • 25
1
vote
1 answer

SpringXD/Spring Integration: Using different versions of spring-integration-kafka for producer and consumer

I have the following configuration: Spring-integration-kafka 1.3.1.RELEASE I have a custom kafka-sink and a custom kafka-source The configuration I want to have: I'd like to still using Spring-integration-kafka 1.3.1.RELEASE with my custom…
Columb1a
  • 463
  • 2
  • 11
  • 25
1
vote
1 answer

SpringXD - It is possible to scan different dirs via File source?

I've these dirs |- Dad Dir |--- Son Dir 1 |---- file1.txt |---- file2.txt |--- Son Dir 2 |---- file3.txt it is possibile with a single stream, using the File Source, extrapolate the contents of…
strano
  • 171
  • 12
1
vote
1 answer

Spring XD - Logical operators in SpEL expression and filters

I would to create a filter like this file --outputType=text/plain --dir=someDir --mode=lines | filter --expression='payload.contains(\"request url\") AND payload.contains(\"request method\")' | log So, i would to visualize on the console the lines…
strano
  • 171
  • 12
1
vote
1 answer

Spring XD - Could not resolve placeholder 'xd.module.sequence'

I am dealing with an issue similar to the following posts: Spring -XD Stream deployment failure issue in spring xd cluster when deploying my module Also, saw that a JIRA ticket was opened: https://jira.spring.io/browse/XD-3727 I created a basic…
Shawn Stewart
  • 13
  • 3
1
vote
1 answer

How to add new header from spring-module.xml

I need to add header from spring-module.xml. I am getting below response headers: file_name: sample.txt content_type: text/plain Payload: { "json":{ "type":"abcd","value":"1234" }} In the header I need to add another field like…
Java Learing
  • 269
  • 3
  • 11
  • 26
1
vote
1 answer

java.lang.NoClassDefFoundError: com/sap/conn/jco/JCoException Error

java.lang.NoClassDefFoundError: com/sap/conn/jco/JCoException at testSAP.TestSAP.execute(TestSAP.java:41) ~[na:na] at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406)…
Suman Goswami
  • 33
  • 1
  • 4
1
vote
0 answers

Using custom SpEL method in spring XD stream definition

My idea is to create custom SpEL method and register that in a spring XD module. I have following in my spring XD module XML
CHEK D
  • 63
  • 1
  • 7
1
vote
1 answer

Can spring xd work with an external spark cluster in yarn mode?

I have a spark cluster in yarn mode, and some batch and streaming jobs to run. I find that spring xd could process batch and stream jobs, with its embedded spark. My question is, can spring xd use an external spark cluster in yarn mode? How to do…
Lionel
  • 13
  • 2
1
vote
1 answer

Couldn't get Spring XD field-value-counter of a json string that is sent through RabbitMQ

I'm trying to build a Spring XD stream that takes a field-value-counter of a json string. Json string is sent in a message through RabbitMQ queue with header Content-Type of application/json. Spring XD logs the message(Json string) and writes it to…
Gishãn
  • 13
  • 3
1
vote
1 answer

springxd - unable to deserialize [classname] Class not found

I am facing this issue while saving data from ESB JMS Queue to HDFS in avro using springxd. I have created custom processor to parse inbound xml from JMS Queue and convert to avro and save in HDFS using springxd stream. Here is my stream definition…
AJ007
  • 125
  • 2
  • 11
1
vote
1 answer

Spring-xd-batch job locking up

I'm using spring-xd for data ingestion via batch jobs. A good number of jobs run in parallel across 4 containers. Any where between 10 and 40 jobs. Most of these complete in less than a minute. I'm using redis (not rabbitmq) and mysql for data…
olumide adebayo
  • 68
  • 5
1
vote
0 answers

But my deployment is getting failed, due to the exception Could not resolve placeholder 'xd.job.name'

I'm using spring xd to run a batch job. I want to read the file location in job.xml from properties file. job.xml …
sophie
  • 11
  • 1
1 2 3
…
37 38