0

I'm looking for a package to run background tasks in Django.

I came across django-background-tasks but there are compatibility issues with the latest Django version since this package appears to be no longer maintained.

I am calling a function run_job from views.py that runs simulations via matlab engine

def run_job():
   eng = matlab.engine.start_matlab()

   eng.addpath(self.utils_dir)
   eng.addpath(self.inp_dir)

   eng.cd(self.t_dir, nargout=0)
   eng.main([self.data_path], nargout=0)

After uploading a file in the frontend, in the backend, this function is called in views.py. The problem is, currently, I am waiting for the simulation to be complete. This is an issue i.e. the job terminates if the user switches to other tabs. Therefore, I would like to know how to run the simulation in the background such that there is no interruption when the user switches to other tabs of the web page in the front,end.

Suggestions on other packages that I could use to run tasks in the background will be of great help.

EDIT:

In models.py, I've a class

Create your models here.

 class file_upload(models.Model):
    uploader = models.ForeignKey(User, on_delete=models.CASCADE)
    ids = models.AutoField(primary_key=True)
    added_on = models.DateTimeField(auto_now_add=True, null=True)

    file_name = models.CharField(max_length=255)

    # registration
    verification_token = models.CharField(max_length=255, null=True, blank=True, default="")

    # running job
    # running_job = models.CharField(max_length=255)

    # finished jobs
    # finished_jobs = models.CharField(max_length=255)

    # task dict {task_name, task_status}
    task_info = models.TextField(null=True, blank=True)

To the same class, I was trying to add task_info variable to save information about the finished and running tasks.

I'm not very sure if management command should be added to models.py.

Natasha
  • 1,111
  • 5
  • 28
  • 66

2 Answers2

0

You might write a management command that, say, takes a RunRequest instance and generates a RunResults instance when it finishes. The view could then create a RunRequest instance (with a link to the User instance!) and spawn a process completely separate from the Django server to actually do the processing. Having done this it could immediately return to the user, perhaps by redirecting to a MyRunJobs view which polls the user's RunRequests and RunResults and displays status and provides appropriate links to retrieve or display the results for ones that have completed, and (maybe) an estimated time to completion for the ones that haven't.

Management commands provide the necessary environment for code to access Django models and instances through code which isn't running in a View, but from a "command line" (which in this case might be used only by the process-spawning code).

Vague and sketchy I know, but detail would require knowing more about your environment, the matlab processing, etc.

Alternatively there's a very comprehensive (and complex) package for running stuff asynchronously, called Celery. It supports Django. I have virtually no experience with it because I decided it was a sledgehammer, and my problem a thin-shelled nut.

nigel222
  • 7,582
  • 1
  • 14
  • 22
  • Could you please explain a bit more on how to do this `spawn a process completely separate from the Django server to actually do the processing` ? – Natasha Jul 13 '23 at 15:37
  • I currently have `results` view which displays a table of all the jobs that are complete. I'm trying to separate this out into `finished` and `running` jobs but not able to do it since the jobs are not run on a separate process. I would like to start the jobs on a different process (currently send emails to the user once the simulation is complete) and populate the `finished` tasks table. I don't want to use Celery, l'm looking for a light weight solution. – Natasha Jul 13 '23 at 15:54
  • If there is a repo with a sample code on implementing this, could you please direct me to it? Thanks a lot – Natasha Jul 13 '23 at 16:55
  • Please see my edit – Natasha Jul 13 '23 at 17:03
  • @Natasha How to spawn a process will depend on your operating system and environment. You may be able to simply use python https://docs.python.org/3/library/subprocess.html. You may have a "batch job" queue or a task broker/manager. You probably are constrained by management policies. You may even have to do the computation on a different machine (or a supercomputer!) You definitely don't want to be running something taking minutes/hours/days in the context of a thread of a webserver. – nigel222 Jul 14 '23 at 13:54
0

I was dealing with a similar problem. I managed to solve it using an ASGI server, concretely uvicorn: https://docs.djangoproject.com/en/4.2/howto/deployment/asgi/uvicorn/

When running Django with the server, I used async functions https://docs.djangoproject.com/en/4.2/topics/async/

Here is an example how I would start a background job with a view:

import asyncio
import logging
from django.http import HttpResponse

async def view(request):
    loop = asyncio.get_event_loop()
    loop.create_task(background_job())
    return HttpResponse('Background job started.')

async def background_job():
    await asyncio.sleep(10)
    logging.info('Background job finished.')
Lukáš Folwarczný
  • 1
  • 1