Spring XD distributed mode redis configuration

Question

I am trying to configure spring XD in distributed mode and I unfortunately I was unable to do so. I am trying to back the inter-module channels using redis and I have a 3 node redis 3.0 cluster running.

My changes to the configuration are shown below, please let me know if I am missing out anything or do I need to do any thing else to make my spring instance run in distributed mode:

servers.yml file of xd-container is configured like

            spring:
              profiles: container
            xd:
              transport: redis
            embeddedHsql: false
            .
            .
            .
            .
            spring:
              datasource:
                url: jdbc:mysql://mysql-instance01:3306/springxd
                username: springxd
                password: springxd
                driverClassName: com.mysql.jdbc.Driver
                validationQuery: select 1
            .
            .
            .
            spring:
              redis:
               port: 6379
               host: redis-instance01
               pool:
                 maxIdle: 8
                 minIdle: 0
                 maxActive: -1
                 maxWait: 30000
               sentinel:
                 master: mymaster
                 nodes: redis-instance02:6379,redis-instance03:6379

If I run the xd-container with this configuration I get the error stack trace as shown here

And If I remove sentinel, sentinel:master, sentinel:nodes, the container is starting fine but failing when some stream is deployed, the error log is as shown here

But there is absolutely no problem with my redis cluster when trying to connect from java using JedisCluster.

Am I missing something in this configuration? Any thoughts or comments?

Thanks in advance.

score 0 · Answer 1 · answered Aug 04 '16 at 12:31

increase Redis maxclients

My test app works(distributed mode with 3 containers) with confgiuration (one machine):

spring:
  profiles: container
xd:
  transport: redis
---
# Redis properties
spring:
  redis:
    port: 6379
    host: 127.0.0.1
    pool:
      maxIdle: 8
      minIdle: 0
      maxActive: -1
      maxWait: 30000

Redis is mostly single threaded application.

Redis will attempt to persist the data to disk. While redis forks for this process, it still slows everything down.

For distributed mode - need use kafka, for example.

Hope this helps you.

Spring XD distributed mode redis configuration

1 Answers1