3

I am running spring XD on distributed YARN setup.I am using hortonworks data platform with 6 data nodes and 1 name node. and using name node as a client node.I have invoked xd Shell from name node and admin and containers are running on the data node. So when I create the spring XD stream definition as below:

xd> stream create --name filetest --definition "file | log" --deploy

It looks for the /tmp/xd/input/filetest on the data nodes to which i dont have access to. Is this the normal behavior of spring XD ? I think it should look for the location on the node from which i have invoked the XD shell. Could you please help me on this.

TartanLlama
  • 63,752
  • 13
  • 157
  • 193
Priyadarshan
  • 31
  • 3

1 Answers1

0

The containers (regardless of whether they are running on Yarn or not) have no knowledge of where the shell is running.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Thank you for the reply Gary!! I agree that both are independent of each other but unfortunately i do not have access to the data node and also it is not feasible to make the file available directly in to the data node. Is there any way i can point the stream to look for the file locally on the client node i.e. in my case on name node. – Priyadarshan Dec 15 '15 at 14:54
  • No; it's not possible. I am not familiar with yarn, but you might be able to use a TCP source instead and `netcat` the file contents. Or an http source and POST the file contents with e.g. `curl`. – Gary Russell Dec 15 '15 at 15:47