0

I want to trigger some tasks on specific time by user settings. For example, if user set 4:00 PM then I would run a task at 4:00PM It could be handle in Celery with countdown and eta. But my broker prefer is Kafka. Is there any alternative of Celery countdown and eta ?

The code in Celery is below:

result = add.apply_async((2, 2), countdown=3)

I expect not use Celery, and must use Kafka

Quyen Nguyen
  • 51
  • 8
  • 1
    Faust is made for stream (real-time data) processing so ETA and countdown do not really make sense there... – DejanLekic Oct 29 '19 at 10:54

2 Answers2

2

Faust does have a crontab trigger for kicking off processes. The example below shows a simplistic implementation using crontab that will run jobs every minute:

import faust
import asyncio

app = faust.App(
        'some_print_step',
        broker="kafka://localhost:9092",
    )

@app.crontab("* * * * *")
async def run_every_min():
    print("This process will be triggered every minute.")


@app.crontab('0 18 * * *')
async def run_everyday_at_6pm:
    print('This process will be triggered every day at 6pm.')

You can find more crontab documentation here:

Faust crontab and timer documentation

meherrr
  • 45
  • 2
  • 8
2

If I'm understanding your response to @meherrr correctly then this might be the way to go. using asyncio.sleep() and passing the delay time in with the message could achieve the same kind of behavior as described here: https://docs.celeryproject.org/en/latest/userguide/calling.html#eta-and-countdown

But it's not like a built in feature.

@app.agent(some_topic, concurrency=100)
async def do_something_later(things_to_do):
    async for thing in things_to_do:
        delay_by = thing.time_to_wait
        await asyncio.sleep(delay_by)
        result = do_the_thing_to_the(thing)
        yield result
BWStearns
  • 2,567
  • 2
  • 19
  • 33
  • 1
    If you're using the wait in order to let something else happen before proceeding (which is the only reason I can think of to do this) then you might want to restructure the program to either use some blocking code within the process or have the thing you're waiting on publish a message to kafka when it's done. – BWStearns Nov 21 '19 at 18:27