1

I am writing a flask app that will count page views from multiple websites. I've decided to use Redis and Redispy but I am having a hard time deciding how to structure. Originally I tried to have a something like this 

redis.set("date:YYYYMMDD:site:sitename", 1)

I want to be able to query by date, or by site name and display the count value. I tried to use .keys to query by date or by site name but the REDIS docs say to avoid using keys.
So then I thought maybe I can use redis hashes:

redis.hset("date:YYYYMMDD", "site", "sitename")
redis.hset("counter", 1)

Ultimately I want to be able run reports on site counters by site name, by date, or display all values from all dates. I can't seem to find the right structure to allow me get what I want.

Any suggestions would be greatly appreciated! I have not used REDIS before.

Ptrkcon
  • 1,455
  • 3
  • 16
  • 32

2 Answers2

1

For overall incrementing counters such as described, a structure along the lines of this answer would be a proven way of doing it. The answer given has more detail son the use of sorted sets for time series data such as this. It even has methods for rolling data up (maybe you also want to report on monthly data).

If, as your examples appear to indicate, your resolution is daily then a structure which stores data in a sorted set with the site/domain name in the key and uses a daily timestamp and [ZINCRBY][2] would serve you well.

With regards to keys ... don't. Simply don't.

As an alternative, you sets to store the keys when using domains/sitenames as your identifier. Whenever you run a ZINCRBY for a site/domain name also do an SADD operation on an "index key" such as sites-with-traffic:YYYY:MM:DD. That way you can pull the set of unique sites for that day and build your queries for each one from there.

In your description you say you may want to report on given days. The way to do this with Redis is to simply query for the key data would be in if it were there. If you get nothing back, the traffic for that period is 0. Don't query when you have a set of parameters - use them. The only one of these query patterns you listed which would be possibly unknown would be the aforementioned site/domain name and adding those to a set solves that need.

Community
  • 1
  • 1
The Real Bill
  • 14,884
  • 8
  • 37
  • 39
0

For counting, you can use incr/decr command

http://redis.io/commands/incr

When you receive access, increment counter like this:

  > incr counter:site2:20131101
  (integer) 1
  > incr counter:site1:20131103
  (integer) 1
  > incr counter:site1:20131103
  (integer) 2
  > incr counter:site2:20131103
  (integer) 1

When you query counters, first get all keys then sum up all counters in your app.

  > keys counter:site2:* # query by site
  1) "counter:site2:20131101"
  2) "counter:site2:20131103"
  > get "counter:site2:20131101"
  "1"
  > get "counter:site2:20131103"
  "1"
  > keys counter:*:20131103 # query by date
  1) "counter:site1:20131103"
  2) "counter:site2:20131103"
  ...
quiver
  • 4,230
  • 1
  • 22
  • 20
  • Hey Quiver, thanks for the response! I was worried about the disclaimer on the Redis page about using KEYS in a production environment. http://redis.io/commands/keys. I am not sure if there is another method that I should use instead of KEYS. Is there any benefit to naming the set counter:... ? – Ptrkcon Nov 03 '13 at 16:57
  • 1
    I haven't tried `keys` for large datasets, so i don't know how well it scales. Also, iterable operation is in development but it's not backported to stable release. http://antirez.com/news/63 This may make counting more easier in the future. Redis fits best for key-value operation. If you need fine-control over reporting, you should store datasets in other databases, in my opinion. – quiver Nov 04 '13 at 12:07
  • Thanks for that link. I'll keep an eye out for that feature, it will definitely make things easier. In terms of reporting, I would just like to query by date, date range, and site name. I think this can all be done currently with keys. – Ptrkcon Nov 04 '13 at 15:46