3

I deployed an distributed configuration on my local machine.

1 admin 1 container 1 zookeeper 1 hsql 1 redis

I have the follwing streams:

stream create --name activityLog --definition "tail --name=/home/bruno/randomName | log"
stream create --name activityLogCounterTap --definition "tap:stream:activityLog > json-to-tuple | filter --expression=payload.reportable.equals(true) | counter --name=activitylogcount"
stream create --name activityLogEventTypeCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=status --name=status"
stream create --name activityLogClientTypeCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=clientType --name=clientType"
stream create --name activityLogIndexCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=index --name=index"
stream create --name activityLogCustomerCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=customer --name=customer"
stream create --name activityLogChannelCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=channel --name=channel"
stream create --name activityLogStrategyOnlineCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=strategyOnline --name=strategyOnline"
stream create --name activityLogStrategyOfflineCounterTap --definition "tap:stream:activityLogCounterTap.filter > field-value-counter --fieldName=strategyOffline --name=strategyOffline"

I want to deploy using the following deployment descriptor:

stream deploy --name activityLog --properties "module.*.count=0"
stream deploy --name activityLogCounterTap --properties "module.*.count=0"
stream deploy --name activityLogEventTypeCounterTap --properties "module.*.count=0"
stream deploy --name activityLogClientTypeCounterTap --properties "module.*.count=0"
stream deploy --name activityLogIndexCounterTap --properties "module.*.count=0"
stream deploy --name activityLogCustomerCounterTap --properties "module.*.count=0"
stream deploy --name activityLogChannelCounterTap --properties "module.*.count=0"
stream deploy --name activityLogStrategyOnlineCounterTap --properties "module.*.count=0"
stream deploy --name activityLogStrategyOfflineCounterTap --properties "module.*.count=0"

The messages are json objects.

The idea is to have an arbitraty number for containers consuming messages from apache kafka however for the example a simple tail is used.

I deployed one by one and I can see that the performance using redis as transport degrades a lot after the 4th stream is deployed. I observe using visual vm that there are some sort of race conditions in inbound.*redis:queue-inbound-channel-adapter1. If I deploy more streams(taps) it becomes unusable.

(I tried to add an image of threads view from Visual VM here however I need 10 reputation :s)

It seems to be locking here:

"inbound.activityLog.0-redis:queue-inbound-channel-adapter1" prio=10 tid=0x00007fe3581a1800 nid=0x29c7 waiting on condition [0x00007fe3d28de000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000788b48d48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:524)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:438)
    at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361)
    at redis.clients.util.Pool.getResource(Pool.java:40)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:90)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:143)
    at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:41)
    at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128)
    at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91)
    at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:78)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:177)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)
    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84)
    at org.springframework.data.redis.core.DefaultListOperations.rightPop(DefaultListOperations.java:151)
    at org.springframework.data.redis.core.DefaultBoundListOperations.rightPop(DefaultBoundListOperations.java:92)
    at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.popMessageAndSend(RedisQueueMessageDrivenEndpoint.java:177)
    at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.access$300(RedisQueueMessageDrivenEndpoint.java:50)
    at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint$ListenerTask.run(RedisQueueMessageDrivenEndpoint.java:290)
    at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

If less than 4 streams are deployed the waiting condition never happens:

"inbound.activityLog.0-redis:queue-inbound-channel-adapter1" prio=10 tid=0x00007ff5101f6800 nid=0x352c runnable [0x00007ff5794a4000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:152)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at java.net.SocketInputStream.read(SocketInputStream.java:108)
    at redis.clients.util.RedisInputStream.fill(RedisInputStream.java:109)
    at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:45)
    at redis.clients.jedis.Protocol.process(Protocol.java:120)
    at redis.clients.jedis.Protocol.read(Protocol.java:191)
    at redis.clients.jedis.Connection.getBinaryMultiBulkReply(Connection.java:212)
    at redis.clients.jedis.BinaryJedis.brpop(BinaryJedis.java:2068)
    at org.springframework.data.redis.connection.jedis.JedisConnection.bRPop(JedisConnection.java:1514)
    at org.springframework.data.redis.core.DefaultListOperations$12.inRedis(DefaultListOperations.java:154)
    at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:50)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:190)
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:152)
    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:84)
    at org.springframework.data.redis.core.DefaultListOperations.rightPop(DefaultListOperations.java:151)
    at org.springframework.data.redis.core.DefaultBoundListOperations.rightPop(DefaultBoundListOperations.java:92)
    at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.popMessageAndSend(RedisQueueMessageDrivenEndpoint.java:177)
    at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint.access$300(RedisQueueMessageDrivenEndpoint.java:50)
    at org.springframework.integration.redis.inbound.RedisQueueMessageDrivenEndpoint$ListenerTask.run(RedisQueueMessageDrivenEndpoint.java:290)
    at org.springframework.integration.util.ErrorHandlingTaskExecutor$1.run(ErrorHandlingTaskExecutor.java:52)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - None

  • My first question is why isn't spring-xd using redis as transport being able to handle more taps. I tried with rabbitMq and it works.

  • My second question is why is it using redis in the first place hence I deployed with in memory configuration. Again using in memory configuration but this time using rabbitMQ I can trace messages being passed which shouldn't happen I think.

--edited--

So regarding the first question I figured out maxTotal parameter in org.apache.commons.pool2.GenericObjectPool default is 8. I went in debug mode and altered this value in runtime (no limit = -1) and that seems to solve de problem. To be able to configure it to deploy with that hack I needed to create an extension (from documentation) and override the following beans:

<bean id="messageBus" class="org.springframework.xd.dirt.integration.redis.RedisMessageBus">
            <constructor-arg ref="redisConnectionFactory2" />
            <constructor-arg ref="codec"/>
            <property name="defaultBackOffInitialInterval" value="${xd.messagebus.redis.default.backOffInitialInterval}" />
            <property name="defaultBackOffMaxInterval" value="${xd.messagebus.redis.default.backOffMaxInterval}" />
            <property name="defaultBackOffMultiplier" value="${xd.messagebus.redis.default.backOffMultiplier}" />
            <property name="defaultConcurrency" value="${xd.messagebus.redis.default.concurrency}" />
            <property name="defaultMaxAttempts" value="${xd.messagebus.redis.default.maxAttempts}" />
    </bean>


<bean id="counterRepository"
            class="org.springframework.xd.analytics.metrics.redis.RedisCounterRepository">
            <constructor-arg ref="redisConnectionFactory2" />
    </bean>

    <bean id="fieldValueCounterRepository"
            class="org.springframework.xd.analytics.metrics.redis.RedisFieldValueCounterRepository">
            <constructor-arg ref="redisConnectionFactory2" />
    </bean>

    <bean id="gaugeRepository"
            class="org.springframework.xd.analytics.metrics.redis.RedisGaugeRepository">
            <constructor-arg ref="redisConnectionFactory2" />
    </bean>

    <bean id="richGaugeRepository"
            class="org.springframework.xd.analytics.metrics.redis.RedisRichGaugeRepository">
            <constructor-arg ref="redisConnectionFactory2" />
    </bean>

    <bean id="aggregateCounterRepository"
            class="org.springframework.xd.analytics.metrics.redis.RedisAggregateCounterRepository">
            <constructor-arg ref="redisConnectionFactory2" />
    </bean>

<bean id="redisConnectionFactory2" lazy-init="false"
      class="org.springframework.data.redis.connection.jedis.JedisConnectionFactory">
    <property name="hostName" value="${spring.redis.host}" />
    <property name="port" value="${spring.redis.port}" />
    <property name="password" value="" />
    <property name="poolConfig" ref="jedisPoolConfig"/>
</bean>

<bean id="jedisPoolConfig" class="redis.clients.jedis.JedisPoolConfig">
    <property name="maxTotal" value="-1"/>
</bean>

I tried at first just to define a redisConnectionFactory bean that would override the default one however it didn't work that way.

I'm not sure if the solution is to make the maxTotal configurable within servers.yml to allow the increase of that value or if somehow there is a bug in which the poolables are not being released. Anyway I will open a jira an hope this helps someone not going through all the work I went :-)

jbrunodomingues
  • 33
  • 4
  • Thanks for your post. This is a fairly complex scenario with nested taps, etc. It's possibly a bug or something that requires further analysis. I would suggest creating an issue at https://jira.spring.io/browse/XD. Regarding your second question: The default transport and analytics are Redis in distributed node, so XD should not be using Redis at all at runtime if configured as you describe. Please double check your configuration and provide more details if you still see this behavior. – dturanski Sep 15 '14 at 18:15
  • Thank you for the reply. This doesn't happen only with nested taps. It also happens if I tap only the first stream and add the filters to every tap instead of taping the first tap. I will take your suggestion and open a jira. – jbrunodomingues Sep 16 '14 at 08:15
  • Regarding the second part I'm running in distributed mode. I tried with both redis and rabbit for transport. However in the deployment descriptor I launch with "module.*.count=0" which, according to the documentation, should launch the streams in every container and forcing the transport to be in memory. Taking as example the rabbit test I can see, using tracing, that the messages are going through rabbit. I also see in the runtime modules that its supposed to be using in memory hence "producer.directBindingAllowed=true" is shown. – jbrunodomingues Sep 16 '14 at 08:16

1 Answers1

0

The problem here is that Spring Boot (which Spring XD is based on) by default sets the redis connection pool size to 8 (see spring.redis.pool.max-active at https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html). This is why the GenericObjectPool size is 8 as mentioned in the question (Jedis uses GenericObjectPool).

Also as shown in the second stack trace Spring XD can take a connection from the pool and block on it until it receives a message. With only 8 connections in the pool, this can soon lead to very poor performance if multiple streams / taps / queues etc are deployed in the same XD container.

However its easy to change the default pool size without the hack mentioned in the question - the max-active parameter is already configurable in servers.yml (since this is just based on the spring boot configuration) eg 

# Redis properties
spring:
  redis:
   port: 6379
   host: 127.0.0.1
   pool:
      maxActive: 100
      maxIdle: 100
#   sentinel:
#     master: mymaster
#     nodes: 127.0.0.1:26379,127.0.0.1:26380,127.0.0.1:26381

This isn't currently well documented in Spring XD but the defaults will be documented in servers.yml in the next release, see https://jira.spring.io/browse/XD-3733

David Geary
  • 1,756
  • 2
  • 14
  • 23