0

I am working on Spring XD and GemFire XD. I want to understand how Spring XD's distributed environment works. I know spring xd uses either redis or rabittmq as the transport.

I am clear about this, I have install spring xd and rabittmq on one machine. I changed the redis.properties file and added hostnames.

Do I need to install spring xd on all the machines? If so, after installing, how to bring those up.

On the master machine, I will do ./xd-admin and ./xd-container

How do you start up the nodes (spring xd instances/workers) so that they can listen for instructions from xd-admin?

Please help me on this.

Thanks, -Suyodhan

user152468
  • 3,202
  • 6
  • 27
  • 57
Madhu Mohan Kommu
  • 335
  • 2
  • 13

3 Answers3

1

Redis is used for analytics as only supported platform. For transport, you need either Redis or Rabbit.

Basically you just need to install Redis and RabbitMQ per their respective documentation. They can be in same or different servers, Ideally you would use their high availability option. For example Redis Sentinal. YOu don't need RabbitMQ unless you want to change the default transport from Redis to Rabbit. Once you install Redis and Rabbit, bring them up and provide their host:port info (and any additional as applicable) to the servers.yml in XD install (in all nodes) and bring up admin and containers. Evrything should work automatically by using zookeeper as the means to manage the distributed runtime.

SathiyaS
  • 246
  • 3
  • 11
0

If you use Spring XD in distributed mode, I assume you have set up zookeeper as well. (If not check this http://docs.spring.io/spring-xd/docs/1.0.0.M7/reference/html/#_setting_up_zookeeper )

Admin and Container instances register themselves with Zookeeper as they come up. Admin queries zookeeper for available containers and assign tasks like deploying modules. Zookeeper is the trick behind Distributed mode.

Hope this helps.

SathiyaS
  • 246
  • 3
  • 11
  • If this is the case, what is the use of redis/rabittmq. I want to install spring xd in distributed mode. As you suggested, I will install Zookeeper in all the nodes and will let spring xd master know this configuration. I need to install spring xd in all the nodes and run xd-container in the nodes. So finally, in one node (spring xd master) xd-admin, xd-container, in rest of the nodes xd-container services I will start. all these node information I will give to the spring xd master(where xd-admin running) through zookeeper. In this case where is redis/rabittmq usage? Please help me. – Madhu Mohan Kommu Jul 05 '14 at 12:52
0

You will install Spring xd one time on one machine, Spring XD will be connected to your hdfs distributed scaled out environment. You need to start the followings: 1. redis or rappitMQ in your case 2. hsqldb server 3. container 4. admin

when you start spring xd, you need to register the name node firstly using the command: hadoop config fs --name hdfs://serverip:8020

then you can use any module defined in spring xd (using stream or batch) by specifying its parameters directly without specifying those in the server.yml file.

Moha.

Mohammad Abu Shaera
  • 21
  • 5