Limiting the number of threads to be executed

Question

I am trying to limit the number of threads that must be executed simultaneously or to separate the elements of the array in a certain amount and send them separately (splice) but I have no idea where and how to start. The code is executing the Threads quickly and this is hindering many.

My code:

file = File.open('lista.txt').read
file.gsub!(/\r\n?/, "")
arr = []

file.each_line do |log|
   arr << Thread.new{

        puts login = Utils.req("GET","http://httpbin.org/GET",{
            'Host': 'api.httpbin.org',
            'content-type': 'application/json',
            'accept': '*/*',
        },
        false)
    }
end

arr.each(&:join)

score 3 · Answer 1 · answered Oct 09 '20 at 06:40

If you want to limit how many threads are running concurrently, use a thread pool. This creates a "pool" of worker threads and limits the number of threads running at the same time. You can write your own, or use a gem like celluloid.

class Whatever
  include Celluloid

  def login_request
    Utils.req(
      "POST",
      "http://httpbin.org/post",
      {
        'Host': 'api.httpbin.org',
        'content-type': 'application/json',
        'accept': '*/*',
      },
      false
    )
  end
end

# Only 5 threads will run at a time.
pool = Whatever.pool(size: 5)

file.each_line do |log|
  split = log.split("|")
  id = split[0]
  number = split[1]

  # If there's a free working in the pool, this will execute immediately.
  # If not, it will wait until there is a free worker.
  pool.login_request
end

I used your code as an example and code appears to run 1 thread at a time, synchronously — JULIO CESAR, Oct 09 '20 at 13:32

Stefan · Answer 2 · 2020-10-09T11:59:34.367

You could use a thread pool. Here's a very basic one using Queue:

queue = Queue.new

pool = Array.new(5) do
  Thread.new do
    loop do
      line = queue.pop
      break if line == :stop

      # do something with line, e.g.
      # id, number = line.split('|')
      # Utils.req(...)
    end
  end
end

It creates 5 threads and stores them in an array for later reference. Each thread runs a loop, calling Queue#pop to fetch a line from the queue. If the queue is empty, pop will suspend the thread until data becomes available. So initially, the threads will just sit there waiting for work to come:

queue.num_waiting  #=> 5
pool.map(&:status) #=> ["sleep", "sleep", "sleep", "sleep", "sleep"]

Once the thread has retrieved a line, it will process it (to be implemented) and fetch a new one (or fall asleep again if there is none). If the line happens to be the symbol :stop, the thread will break the loop and terminate. (a very pragmatic approach)

To fill the queue, we can use a simple loop in our main thread:

file.each_line { |line| queue << line }

Afterwards, we push 5 :stop symbols to the end of the queue and then wait for the threads to finish:

pool.each { queue << :stop }
pool.each(&:join)

At this point, no threads are waiting for the queue – they all terminated normally:

queue.num_waiting  #=> 0
pool.map(&:status) #=> [false, false, false, false, false]

Note that Queue isn't limited to strings or symbols. You can push any Ruby object to it. So instead of pushing the raw line, you could push the pre-processed data:

file.each_line do |line|
  id, number = line.split('|')
  queue << [id, number]
end

This way, your threads don't have to know about the log format. They can just work on id and number:

loop do
  value = queue.pop
  break if value == :stop

  id, number = value
  # do something with id and number
end

Instead of writing your own thread pool, you could of course use one of the various concurrency gems.

Limiting the number of threads to be executed

2 Answers2