1

pub.py

import redis
import datetime
import time
import json
import sys

import threading
import gevent
from gevent import monkey
monkey.patch_all()

def main(chan):
    redis_host = '10.235.13.29'
    r = redis.client.StrictRedis(host=redis_host, port=6379)
    while True:
        def getpkg():
            package = {'time': time.time(),
                        'signature' : 'content'
                      }

            return package

        #test 2: complex data
        now = json.dumps(getpkg())

        # send it
        r.publish(chan, now)
        print 'Sending {0}'.format(now)
        print 'data type is %s' % type(now)
        time.sleep(1)

def zerg_rush(n):
    for x in range(n):
        t = threading.Thread(target=main, args=(x,))
        t.setDaemon(True)
        t.start()

if __name__ == '__main__':
    num_of_chan = 10
    zerg_rush(num_of_chan)
    cnt = 0
    stop_cnt = 21
    while True:
        print 'Waiting'
        cnt += 1
        if cnt == stop_cnt:
            sys.exit(0)
        time.sleep(30)

sub.py

import redis
import threading
import time
import json
import gevent
from gevent import monkey
monkey.patch_all()

def callback(ind):
    redis_host = '10.235.13.29'
    r = redis.client.StrictRedis(host=redis_host, port=6379)
    sub = r.pubsub()
    sub.subscribe(str(ind))
    start = False
    avg = 0
    tot = 0
    sum = 0
    while True:
        for m in sub.listen():
            if not start:
                start = True
                continue
            got_time = time.time()
            decoded = json.loads(m['data'])
            sent_time = float(decoded['time'])
            dur = got_time - sent_time
            tot += 1
            sum += dur
            avg = sum / tot

            print decoded #'Recieved: {0}'.format(m['data'])
            file_name = 'logs/sub_%s' % ind
            f = open(file_name, 'a')
            f.write('processing no. %s' % tot)
            f.write('it took %s' % dur)
            f.write('current avg: %s\n' % avg)
            f.close()

def zerg_rush(n):
    for x in range(n):
        t = threading.Thread(target=callback, args=(x,))
        t.setDaemon(True)
        t.start()

def main():
    num_of_chan = 10
    zerg_rush(num_of_chan)
    while True:
        print 'Waiting'
        time.sleep(30)

if __name__ == '__main__':
    main()

I am testing redis pubsub to replace the use of rsh to communicate with remote boxes.
One of the things I have tested for was the number of channels affecting latency of publish and pubsub.listen().

Test: One publisher and one subscriber per channel (publisher publish every one second). Incremented the number of channels from and observed the latency (The duration from the moment publisher publish a message to the moment subscriber got the message via listen)

num of chan--------------avg latency in seconds
10:----------------------------------0.004453
50:----------------------------------0.005246
100:---------------------------------0.0155
200:---------------------------------0.0221
300:---------------------------------0.0621

Note: tested on 2 CPU + 4GB RAM + 1 NICs RHEL6.4 VM.

  1. What can I do to maintain low latency with high number of channels?

  2. Redis is single-threaded so increasing more cpus wont help. maybe more RAM? if so, how much more?

  3. Anything I can do code-wise or bottleneck is in Redis itself?

  4. Maybe the limitation comes from the way my test codes are written with threading?

EDIT: Redis Cluster vs ZeroMQ in Pub/Sub, for horizontally scaled distributed systems

Accepted answer says "You want to minimize latency, I guess. The number of channels is irrelevant. The key factors are the number of publishers and number of subscribers, message size, number of messages per second per publisher, number of messages received by each subscriber, roughly. ZeroMQ can do several million small messages per second from one node to another; your bottleneck will be the network long before it's the software. Most high-volume pubsub architectures therefore use something like PGM multicast, which ZeroMQ supports."

From my testings, i dont know if this is true. (The claim that the number of channels is irrelevant)
For example, i did a testing.

1) One channel. 100 publishers publishing to a channel with 1 subscriber listening. Publisher publishing one second at a time. latency was 0.00965 seconds
2) Same testing except 1000 publishers. latency was 0.00808 seconds

Now during my channel testing:
300 channels with 1 pub - 1 sub resulted in 0.0621 and this is only 600 connections which is less than above testing yet significantly slow in latency

Community
  • 1
  • 1
ealeon
  • 12,074
  • 24
  • 92
  • 173
  • http://highscalability.com/blog/2014/4/28/how-disqus-went-realtime-with-165k-messages-per-second-and-l.html suggests another approach for pub/sub and still keeps using Redis with zsets. In any case, 62ms is not a bad latency I think. – Niloct Oct 12 '14 at 16:27
  • @Niloct thanks for the article and yeah i can live with 62ms latency. However, from my testings, degradation in latency is directly proportionate to the num of channel. I dont know Redis internals to explain that. and I am worried if i increase more channels, latency will get worse to unacceptable level. At that point, what do i do to mitigate that? – ealeon Oct 12 '14 at 17:40
  • I would simulate the worst environment you wish to stand, and check the latency. In the link there is another pub/sub from nginx which you could try too. – Niloct Oct 14 '14 at 16:34
  • You can also contact the Redis' author, he's quite a nice guy and may share his insight on this. – Niloct Oct 14 '14 at 16:35
  • 1
    Andy McCurdy (redis-py's author) suggests that this has nothing to do with Redis itself but rather the choice of the test code's threading model: https://groups.google.com/d/msg/redis-db/WY4djaIYbTs/opcxeXlVi4wJ – Itamar Haber Oct 15 '14 at 10:15
  • I would go so far as to say that this isn't even a suggestion, but a statement of fact that this is related to Python threads being used to benchmark causing the latency issues. – Josiah Oct 16 '14 at 05:41

0 Answers0