0

There are 1 million customers. I need to send them in batches to a third-party service, get a response and process. the same code for 500_000 customers works.

thread_pool = Concurrent::FixedThreadPool.new(100)

executors = customers.each_slice(150).map do |batch|
  Concurrent::Future.execute(executor: thread_pool) do
    # hard work
  end
end

This code does not work, crashes or with an error can't create Thread: Resource temporarily unavailable. Or crashes with an error 137 (by memory) from k8s, when I work in the terminal.

How can I optimize my work with concurrent?

evans
  • 549
  • 7
  • 22
  • 1
    Can you elaborate on the problem you are trying to solve? What do you mean by "send customers to a service"? Like redirecting on a website? Or do you send out emails? Do you load all customers into memory at once and therefore run out of memory? Did you try `find_each` or `in_batches`? – spickermann Mar 17 '22 at 07:13
  • customers are a array of hashes `[{email: '', msisdn: '', values: ''}, ...]`. 150 clients is the third-party service limit. I use `each_slice` – evans Mar 17 '22 at 07:54

0 Answers0