Using EventMachine to setup infinite loop to generate random data

Question

I'm trying to setup an automated stress test, and feed the randomly generated data into Redis, and then have the consumer (as another component read from Redis) to process the random data.

To simulate the randomly generated data as close to real-world timing, I decided to put in an infinite loop, and using EventMachine to handle the synchrony. I'm not sure what I'm doing with the EventMachine, but I heard it would be much better than keep spawning new threads and taking blocking the main process. Am I doing this right?

EventMachine.synchrony do

  @redis = EM::Hiredis.connect

  # hit Control + C to stop
  Signal.trap("INT")  { EventMachine.stop }
  Signal.trap("TERM") { EventMachine.stop }

  trip_id = 0
  loop do
    longitude, latitude = [0.0, 0.0]
    for i in 0..100
      # now loop
      longitude, latitude = random_coordinates

      sleep 0.05 # sleep a bit to simulate the real world 
    end

    puts "this trip #{trip_id} | #{longitude} : #{latitude}"
    trip_id += 1

    redis.set('trip_id', trip_id)
  end
end

EDITED 1

So I end up doing it with a thread pool

require "thread/pool"
require "json"
require "redis"

require 'thread_safe'

def publish_on_redis(hash)
  @redis ||= Redis.new
  @redis.publish("trips", hash.to_json)
end

# Using thread-safe data structure
@trip_id_list = ThreadSafe::Array.new()
TRIP.times {|n| @trip_id_list[n] = n}
@trip_id_list.shuffle!

pool = Thread.pool(TRIP)
loop do
  @trip_id_list.each do |trip_id|
    pool.process {
      longitude, latitude = [0.0, 0.0]
      # Everything has to be UTC timestamp, everyone is on the same timezone
      start = Time.now.utc
      while((Time.now.utc - start) < @total_duration)
        longitude, latitude = random_coordinates_radius
        publish_on_redis({:id => trip_id, :time => Time.now, :longitude => longitude, :latitude => latitude})
        sleep 1 # sleep at each run some amount of times
      end
      publish_on_redis({:id => trip_id, :time => Time.now, :longitude => longitude, :latitude => latitude})
      @trip_id_list.delete(trip_id)
    }
  end

  break # will have to exit the infinite loop
end
pool.shutdown

score 0 · Answer 1 · answered Mar 27 '14 at 22:33

Generating random events like you are doing here, seems about right.

However, you are not stressing redis at all, you are only doing a simple 'set' which will overwrite the key with new data if it exists.

I suggest using a pipeline which you fill with quite a lot of data, which you execute+await in the testing block (with a sleep).

Move the sleep statement outside of your data preparation code.

Hope this helps, TW

Using EventMachine to setup infinite loop to generate random data

1 Answers1