1

I have a Django app that receives sensor data. This data is then processed and written to influxDB using the influxdb-client-python library. I would like to write the data in an asynchronous manner and thus return a response to the producer of the data before it is actually written to the database.

However, once I send this response I can no longer afford to lose this data. Since I can never be sure that the server will in fact be able to write the data to influxDB, I was thinking about first writing it to a file and returning a response after this is successful (similar to a WAL). This introduces new problems like making sure the WAL is actually written to a file and most importantly, making sure it is thread-safe since the server may be handling multiple requests concurrently.

Is there any better way of doing this, maybe built-in in Django?

Willem Govaerts
  • 80
  • 7

1 Answers1

1

That sounds like a queue, background tasks, etc. You are right, it just displaces the issue, but queues are highly reliable as well.

The standard libs for doing this with Django and/or Rest Framework are:

  • Celery - Full featured, mature, pluggable queue backends
  • Django RQ - Simpler, uses only redis

Celery is probably the right starting point here, since it lets you use a "real" queue backend, but Django RQ+redis will also work if there isn't a ton of load.

Without knowing anything more about your app or architecture, its hard to say more. There are a lot of queuing systems (Rabbit, ZeroMQ, AWS SQS, Google's, etc). You can also look into building the queue+processor using, for example, AWS SQS and AWS Lambda Functions (or google versions).

Andrew
  • 8,322
  • 2
  • 47
  • 70