Handling Large Dataset with Redis

Asked Dec 23 '13 at 10:36

Active Dec 23 '13 at 10:36

Viewed 467 times

I am working on crawler application and I am using Redis for holding queue of pending urs which my crawler is going to crawl.

There are multiple nodes which running they fetch URL to crawl from Redis DB and after crawling put new URL (which found on crawled URL pages) in to redis again. I am redis for extremely fast insertion and selection.

My database size grows drastically. After running my crawler for 24 hours I have more than 10 million URL in my Redis (I use url as key and other parameter like depth/crawlTime etc as value). I am using a single machine as Redis server.

I am not clear about the performance of Redis and how I can handle this much data on Redis? or should I move to other databases like MongoDB?

My Machine have 6 GB Ram.

Alex

asked Dec 23 '13 at 10:36

Alex

1,406
2
18
33

What's your memory usage after adding 10 million URLs? (I'm guessing it should be close to 4GB) – Manu Manjunath Dec 23 '13 at 12:17
1

Redis assumes everything will fit in memory. If you think your data will be bigger, then Redis will not suit your needs. – Didier Spezia Dec 23 '13 at 17:02
I checked with Redis Live, and used memory is nearly 7.54 GB. – Alex Dec 24 '13 at 05:07

Handling Large Dataset with Redis

0 Answers0