we are developing data pipeline app using Kafka, storm and redis. Realtime events from different systems will be published to Kafka and storm do the event processing based on rules configured. State is managed in redis.
we have a requirement to implement different WAIT_TIME before processing for different events. we are looking at following options.
we initially looked at storm windowing [sliding or tumbling window] but provides option only to configure fixed intervals. we need varying wait_time based on rules
we are exploring other options of storing the events in a redis cache for varying duration [TTL] and once each events are evicted we need to have a callback back to storm to process it.
Do redis support callback on eviction ? Is there a better way to do this with storm and redis ?
