Storing vector of chars in Redis - containing NUL

Question

I want to store a JPEG image in redis as a single key-value pair. From OpenCV, I get a std::vector<unsigned char> jpeg from imencode()

Now I convert this vector to std::string and SET it with Hiredis. The problem is that the jpeg vector contains NUL characters (ANSII == 0) and the Hiredis SET function receives value.c_str(). .c_str() truncates the string after the first occurrence of NUL, and therefore only this substring is stored in the DB.

My question is: How can I SET and GET a std::vector<unsigned char> (containing NUL) with Hiredis? (Minimizing runtime is critical.)

Here is my code:

// Create vector of uchars, = From CV [Disregard inefficiency here]

std::vector<unsigned char> jpeg;
jpeg.push_back( 'a' );
jpeg.push_back( 'b' );
jpeg.push_back( (unsigned char) 0 );
jpeg.push_back( 'c' );
jpeg.push_back( 'd' );

// Convert to string

std::string word = "";
for (int i=0; i<jpeg.size(); ++i)
{
    word.push_back(jpeg[i]);
}

std::cout << "word = " << word << std::endl;
std::cout << "word.c_str() = " << word.c_str() << std::endl;

// connect redis

std::string hostname = "127.0.0.1";
int port = 6379;
timeval timeout = { 1, 500000 }; // 1.5 seconds
redisContext* context = redisConnectWithTimeout(hostname.c_str(), port, timeout);

// set redis

std::string key = "jpeg";
redisReply* reply = (redisReply *)redisCommand(context, "SET %s %s", key.c_str(), word.c_str() );
freeReplyObject( (void*) reply);

// get redis

reply = (redisReply *)redisCommand(context, "GET %s", key.c_str() );
std::string value = reply->str;
freeReplyObject((void*) reply);
std::cout << "returned value = " << value << std::endl;

// Convert back to vector of uchars (this should be the same as the original jpeg)  [Disregard inefficiency here]

std::vector<unsigned char> jpeg_returned;
for (int i=0; i<value.size(); ++i)
{
    jpeg_returned.push_back(value[i]);
    // std::cout << "value[i] = " << value[i] << std::endl;
}

While `std::string` can store arbitrary data, you really can not use string functions when dealing with binary data (which is what a JPEG image is). Doesn't Redis has some kind of binary blob type? — Some programmer dude, Mar 15 '17 at 04:06
Actually HiRedis lib has binary support and you don't need to convert std::vector to std::string before storing it. Check this: http://stackoverflow.com/questions/26799074/can-we-set-c-int-array-as-a-keys-value-in-redis-by-hiredis — JustRufus, Mar 15 '17 at 04:35
*.c_str() truncates the string after the first occurrence of NUL* -- This is not true. It is not `c_str()` that is truncating the string, it's the functions that you're giving `c_str()` to that stop on the first NULL. [See the documentation on c_str()](http://en.cppreference.com/w/cpp/string/basic_string/c_str). For example, your usage of `cout` will not work like that, as you would need to use `cout.write(word.c_str(), word.size())` to properly output the entire string. — PaulMcKenzie, Mar 15 '17 at 04:39
Yes, jpeg is binary but opencv returns a vec, which is just begging to be stored in redis as a string. The only problem really is the NUL char. I saw link before, the author does not recommend this method. It would also be nice if I could read this jpg from another python app. — user2926577, Mar 15 '17 at 06:42
Sir, can you share your code about storing Opencv image into Redis using C++ and reading it using Python? It will help me a lot, thanks! — ToughMind, Dec 05 '19 at 01:58
Sir, can you share your code about storing Opencv image into Redis using C++ and reading it using Python? It will help me a lot, thanks! — ToughMind, Dec 05 '19 at 01:59

score 2 · Accepted Answer · answered Mar 15 '17 at 05:28

2

Before showing a code I want to warn one more time that storing binary data without proper serialization can be problematic. At least make sure all your servers are the same endianness and has the same sizeof(int).

std::vector<char> v{'A', 'B', '\0', 'C', 'D'};

std::string key = "jpeg";
redisReply* reply = static_cast<redisReply *>( redisCommand(context, "SET %s %b", key.c_str(), v.data(), v.size() ) );
freeReplyObject( reply );

And reading from python.

>>> import redis
>>> r = redis.StrictRedis(host='localhost', port=6379, db=0)
>>> r.get("jpeg")
'AB\x00CD'

answered Mar 15 '17 at 05:28

JustRufus

492
1
5
10

Thank you! I implemented the get functionality, based on your other post: `redisReply* reply = (redisReply *) redisCommand(context, "GET %s", a_key.c_str() ); unsigned char *val = (unsigned char *) malloc( reply->len ); memcpy( val, reply->str, reply->len); std::vector jped_returned(val, val + reply->len); free( val ); freeReplyObject(reply);` Is this the most efficient way? – user2926577 Mar 15 '17 at 07:17
@user2926577 I guess yes. why do you allocate c-style array then copy into std::vector. You should be able to create std::vector jped_returned directly from reply->str and reply->len. – JustRufus Mar 15 '17 at 07:49
I was following your comment `// Here, it is safer to make a copy to be sure memory is properly aligned`, I'm actually not sure why this is advised here. I created a pointer to the array, there shouldn't be any deep copies happening here. Could you post the c++ code for binary GET as you would have optimized it? Thank you – user2926577 Mar 15 '17 at 08:04
@user2926577 you can do something like this: `reply = static_cast(redisCommand(context, "GET %s", key.c_str() ) );` `std::vector read_vector( reply->str, reply->str+reply->len );` – JustRufus Mar 15 '17 at 08:46

Storing vector of chars in Redis - containing NUL

1 Answers1