4

I plan to create a web service which will allow the users to spawn a large number, let's say 3k-5k, of certain types of jobs. There'll be a few different, pre-defined types of jobs.

defmoudle Job1 do
  def do_work1(a, b, c) do

  end
end

defmoudle Job2 do
  def do_work2(a, b, c) do

  end
end

# defmoudle Job10 do
# .......

Each one will take on average 5-10, up to 30 minutes to complete, if this is relevant.

It's important that the user should be able to create them at runtime, view the active ones, view the status of each, optionally pause if possible, view the progress if possible, and terminate the jobs.

Should use DynamicSupervisor for this? How would I start a job? As Task.async(..) ... await(..)? As Task.start(..)? Or anything else?

And I probably wouldn't need to make each job a GenServer, right?

Without overcomplication. Only by the means of OTP. Oban won't suit me in this case.

mondichuk
  • 77
  • 1
  • 5

2 Answers2

2

I don't know why you don't want to use Oban, but if you must, then

  • add max_children: n when you call DynamicSupervisor.init to limit the number of concurrently running jobs.
  • add restart: :temporary for your GenServer.
  • when the user decides to kill a job, use Process.exit(pid, :kill) to brutely kill the process, otherwise it may not be killed instantly.
  • use a registry for naming your GenServer processes so that you can relatively easily find your processes.
  • use another GenServer process to monitor the job.
  • depending on what you mean by "view the status", you may have to implement some sort of telemetry.
  • if you want to pause a running job, then the job should be split into batches. The job should send itself a message after each batch to start the next batch.

Anyway, it's not impossible, but can be pretty hard to implement.

Aetherus
  • 8,720
  • 1
  • 22
  • 36
  • 1) Why GenServer for the jobs? Why not Task? They'll be one-time jobs: create, execute, done. Many times so, but not in a recurring manner. – mondichuk May 04 '23 at 00:27
  • 2) `depending on what you mean by "view the status"` -- a) retrieve all the jobs that are *currently* running. Not those that are simply `children`, but in the state of "running" (doing work). Is it done via `DynamicSupervisor.count_children()` or by other way? – mondichuk May 04 '23 at 00:31
  • b) get the ones that **have been run in the past,** their status, results, errors. Will I have to use the DB for this? – mondichuk May 04 '23 at 00:31
  • 3) `use another GenServer process to monitor the job.` -- why? `DynamicSupervisor` will monitor all the jobs/children – mondichuk May 04 '23 at 00:35
  • 1) because you can't easily implement continuation using Task. 2) children _are_ current running processes. otherwise they are just not there. 3) then you need a database to store those information so that your memory won't leak 4) because a GenServer does things sequentially, meaning that when it's doing its job, it won't handle any message. – Aetherus May 04 '23 at 01:54
  • `1) because you can't easily implement continuation using Task.` -- you mean "pause a running job"? If not for this, `Task` could be used instead? – mondichuk May 04 '23 at 02:55
  • @mondichuk yes. – Aetherus May 04 '23 at 05:56
0

Take a look at Oban. It helps to limit the number of concurrently running jobs. It also helps with job management.

Aetherus
  • 8,720
  • 1
  • 22
  • 36