How to get all hash elements from multiple key using stackexchange.redis

Question

I'm evaluating redis, and an trying to use a lua script to get multiple hash keys from redis in one call.

I can't seem to get it to work. The code below returns an array of items to my c# code, but I can't seem to get at the hash results. the redisResult returned is an array and each element of the array is an empty array. I expected each element to be an array of the hash values.

        int key1 = 1;
        var sb = new StringBuilder();
        var compositeKeys = new List<RedisKey>();
        for (int key2 = 1; key2 < 1000; key2++)
        {
            compositeKeys.Add((RedisKey)GetRedisKey2(key1, key2));
        }


        sb.AppendLine("local collate = function(key)");
        sb.AppendLine("local raw_data = redis.call('HGETALL', key)");
        sb.AppendLine("local data = {}");
        sb.AppendLine("for idx = 1, #raw_data, 2 do");
        sb.AppendLine(" data[raw_data[idx]] = raw_data[idx + 1]");
        sb.AppendLine("end");
        sb.AppendLine("return data;");
        sb.AppendLine("end");

        sb.AppendLine("local data = {}");
        sb.AppendLine("for _, key in pairs(KEYS) do");
        sb.AppendLine("  data[_] = collate(key)");
        sb.AppendLine("end");

        sb.AppendLine("return data");

        var results = _redDb.ScriptEvaluate(sb.ToString(), compositeKeys.ToArray());

EDIT: to better explain what I'm trying to do: This will be used for looking up stock market data, and there are three use cases for getting the data out of the cache.

1. Get a single item by symbol and time
2. Get a range of historical data for a single symbol
3. Get a range of symbols from specific time

Each data point has about 30 fields of data about the stock at that time. I am storing it using a key that combine the symbol and time, and hash fields for each field in my object. The performance I am receiving for a single stock lookup is great, but I am having trouble with looking up a range of values.

I implemented the pipelined approach described in this question: StackExchange.Redis: Batch access for multiple hashes. The performance is OK, but not better than what I get today out of SQL Server. I'd like to benchmark the performance using the LUA scripting approach, but I must be doing something wrong.

Answer 1

Rather than focus on the question title (mainly as I'm not convinced you'll get the performance gains you're looking for), I'll suggest another approach that focuses on the objectives that you added to your question later:

1. Get a single item by symbol and time
2. Get a range of historical data for a single symbol
3. Get a range of symbols from specific time

Given your interest in time based ranges, you may find that sorted sets work better.

A sorted set consists of elements where each element has a score (a double) and a value (any string, e.g. JSON, object serialized using protobuf).

As such you could create a set per stock and add elements to your set where the score is the trade time in milliseconds(*) and the value is some string that describes your trade. e.g.,

ZADD stocks.GOOG 1459956490731 {LastTradePrice:743.16,SomeOtherUsefulInfo:123}

and then to query for all trades within a given time range,

ZRANGE stocks.GOOG <start-time-in-ms> <end-time-in-ms> WITHSCORES

To get a range of symbols for a specific time you'd need an extra set where you add one element per stock that that should be returned, e.g.

ZADD stocks.active 1459956490731 GOOG
ZADD stocks.active 1459956490731 IBM
ZADD stocks.active 1459956490731 AAPL

Hope this helps!

(*) for example,

var startOfTime = new DateTime(2016, 1, 1); 
var tradeTime = DateTime.Now; 
var ms = (tradeTime - startOfTime).TotalMilliseconds;