Stack Overflow
Stack Overflow

Questions tagged [spring-xd]

Spring XD is a unified, distributed, and extensible system for data ingestion, real time analytics, batch processing, and data export. The project’s goal is to simplify the development of big data applications.

Spring XD is a unified, distributed, and extensible system for data ingestion, real time analytics, batch processing, and data export. The project’s goal is to simplify the development of big data applications.

Big data applications share many characteristics with Enterprise Integration and Batch applications. Spring has provided proven solutions for building integration and batch applications for more than 6 years now via the Spring Integration and Spring Batch projects. Spring XD builds upon this foundation and provides a lightweight runtime environment that is easily configured and assembled via a simple DSL.

Click here for more information about Spring XD.

565 questions
4
votes
1 answer

Spring XD dynamic deployment manifest

I have been reading the Spring XD documentation fairly heavily and can't really get to grips with two things I'd like to achieve in relation to Hadoop YARN. Maybe they aren't supported yet or won't ever be supported - possibly because I am missing…
user3880645
  • 41
  • 1
4
votes
1 answer

How to create a job in spring-xd using java config only

Is there any example of using spring xd with pure java config (no xml). I created a simple tasklet, jarred the same (whojob.jar) and put it into the lib folder. The xml config is below. I put this in a file called whojob which is in the modules/job…
user3304825
  • 107
  • 2
  • 12
4
votes
2 answers

Using Greenplum with Spring

Do any of the Spring projects provide a template or utility for inserting into or working with Greenplum? I understand that one approach is, using Spring Batch, to have a tasklet call the Greenplum gpload utility which will then insert a specified…
user1052610
  • 4,440
  • 13
  • 50
  • 101
3
votes
1 answer

An illegal reflective access operation has occurred while setting up spring xd

i was following the guide Spring XD Guide to setup spring xd on window 7. When i tried to start the runtime and the XD shell and entered ./xd-singlenode, it gave me the following warnings and errors: WARNING: An illegal reflective access operation…
Noura
  • 49
  • 1
  • 5
3
votes
1 answer

Spring cloud data flow - Micro services deployment

Team, Currently I am working on spring-xd and using as a runtime container for data analytics and yarn jobs. My questions are 1) Can I leverage the same environment setup which I used for spring-xd? 2) From the documentation,I read that it can be…
Praveen Kumar K S
  • 3,024
  • 1
  • 24
  • 31
3
votes
1 answer

Spring XD filtering inside a processor module

I would like to create a module which will filter messages from the input channel and transform them into something else on the output. I know I can seperate this in two modules (I prefer Java code over scripts) like: @Filter(inputChannel = "input",…
alturkovic
  • 990
  • 8
  • 31
3
votes
1 answer

RabbitMQ - Channel shutdown: connection error (SpringXD closes rabbitmq connections repeatedly.)

I had a terrible night trying to figure out what is going on with RabbitMQ and SpringXD, unfortunately without a success. The problem: SpringXD closes RabbitMQ connections repeatedly, or reports warnings related to the channel cache size. Fragment…
Ivan Prostran
  • 53
  • 1
  • 5
3
votes
0 answers

How to get the Spring XD module class loader?

I am using javassist in a Spring XD module. This breaks since javassist does not use Spring's classpath. I can work around this by supplying the class loader of the module / XD to javassist. For SpringApplications, I know from the link how to do…
Arne
  • 2,624
  • 3
  • 24
  • 45
3
votes
1 answer

File path issue while creating simple spring XD stream

I am running spring XD on distributed YARN setup.I am using hortonworks data platform with 6 data nodes and 1 name node. and using name node as a client node.I have invoked xd Shell from name node and admin and containers are running on the data…
Priyadarshan
  • 31
  • 3
3
votes
1 answer

SpringXD - Error in Gemfire as sink

I am trying to setup a stream in springXD. gemfire is installed in 10.99.40.60 machine and springXD is installed in my LOCAL VM. xd:>stream create --name gemfiredemo --definition "http --port=9090 | gemfire-server…
Amaresh
  • 3,231
  • 7
  • 37
  • 60
3
votes
0 answers

Use Cassandra as Job Repository

I've seen JDBC Drivers out there for Cassandra. Anyone knows if would it be possible to use Cassandra as a Job Repository for Spring XD?
thiago
  • 117
  • 1
  • 7
3
votes
1 answer

Delete a file after all the rows in the file have been processed

I would like to delete a file after all the rows in the file have been processed. My streams look like source (file --fixedDelay=0 --outputType=text/plain --dir=XXX) | splitter --expression=payload.split('\\n') | transform -> filter ->…
Camilo Melani
  • 31
  • 1
3
votes
1 answer

How to call a rest api from spring-xd

I am new to spring-xd and would like to understand is it possible to invoke an external rest api from springxd. My rest api expects a request header for GET requests and request header+message body for POST requests. How do I invoke the URL ? In my…
shankar
  • 33
  • 2
3
votes
1 answer

Spring XD very poor performance when using redis as transport

I deployed an distributed configuration on my local machine. 1 admin 1 container 1 zookeeper 1 hsql 1 redis I have the follwing streams: stream create --name activityLog --definition "tail --name=/home/bruno/randomName | log" stream create --name…
jbrunodomingues
  • 33
  • 4
2
votes
1 answer

reading CSV file from s3 using spark

I am new to Spark. I have a scenario where I need to read and process CSV file from AWS s3. This file is generated on daily basis, so I need to read and process it and dump data into Postgres. I want to process this huge file in parallel to save…
ManojP
  • 6,113
  • 2
  • 37
  • 49
1
2
3
…
37 38