Redis pipelining 200 instructions sent, only 189 answers

Question

I am learning Redis and I am blocked with the pipelining concept, I am trying to send instruction to my redis server

Do to so I using a socket whitch will connect to the redis server I am using.

Here is my code (I am French so some words will be in french)

def send(MESSAGE):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((TCP_IP, TCP_PORT))
    s.send(MESSAGE)
    data = s.recv(BUFFER_SIZE)
    s.close()
    print "Envoi requete PC:", MESSAGE
    return data

And here is the way I am using the pipelining :

instruction ='SET compteur 0'
donnee = instruction.encode('utf-8') + '\x0D\x0A'
print envoie(donnee)
instruction=''
for i in range(200):
    instruction = instruction + 'INCR compteur\r\n'
donnee = instruction.encode('utf-8') + '\x0D\x0A'
print send(donnee)

when I do this, the shell gives me the 200 INCR compteur but it is followed with :

:1
:2
:3
:4
....
:185
:186
:187
:188
:189

Does somebody have an explanation ? Also if I use another instruction for example with a GET compteur, I have only 147 +PONG

Why don't you use a redis adaptor for python instead of defining sockets by hand? https://github.com/andymccurdy/redis-py — Edgar Andrés Margffoy Tuay, Apr 20 '17 at 15:02
I can't use the redis adaptator cause this code will be implemented on a PLC which will not be provided with redis, it will only have python, so I have to do like that, but thanks for the advice — Adrien G, Apr 21 '17 at 07:59

score 0 · Answer 1 · answered Apr 20 '17 at 17:55

0

As Edgar suggested, use the redis-py python library to help manage your redis connections.

Using that:

from redis import StrictRedis

r = StrictRedis(host='localhost', db=1)
pipe = r.pipeline()

for i in xrange(100):
    pipe.ping()

results = pipe.execute()

print len(results)

properly shows 100 PONGs (or True), as expected.

Or a similar test using INCR:

for j in xrange(100):
    pipe.incr("test-incr", 1)

results = pipe.execute()

print len(results)
print results[95:100]

returns:

100
[96, 97, 98, 99, 100]

answered Apr 20 '17 at 17:55

dizzyf

3,263
19
29

1

Thank you dizzyf but I can not use this method, this program will be uploaded in a robot which will not be equiped with redis, it will only "talk" little instruction like I wrote. But thank you for the answers I will use it a different program. – Adrien G Apr 21 '17 at 07:34

iMysak · Accepted Answer · 2017-04-21T00:31:14.253

You can add two commands surround your instraction: MULTI before and EXEC after your instructions. This will guaranty atomicity and get all results of execution your commands. Take a look on transaction documentation.

Please read about pipelining also. There are a lot of useful information here.

In fact I guess your problem is that you could start reading answer before Redis return it.

Although logic of flow submitRequest -> receiveResults looks good, but your code doesn't work in such way because Redis perform operation asynchronically.

In fact, your code reads in next way: send data(instructions) to server-> read some data from socket (result | part of result | nothing?). The problems is we don't wait Redis to complete computation.

I am not good in Python but I guess data = s.recv(BUFFER_SIZE) will read up to 'BUFFER_SIZE' bytes from s if it is present. But if only part of results is present in socket this command return only these part of data. Handling reading from socket is not so trivial operation, as other folks suggest - use already existed libraries - https://redis.io/clients#python

Thank you for the answer, I did not realize that my buffer was so small, I just increased it and it works perfectly. I am reading about the pipelining now it may be a great improvement for my code. — Adrien G, Apr 21 '17 at 07:30

Redis pipelining 200 instructions sent, only 189 answers

2 Answers2