1

I am trying to implement Faust library into my code to consume data from Kafka, run some code asynchronously and return data back to Kafka. I followed the documentation but seem to have made a mistake in my code (or in my understanding of how Faust works) as Faust in my app currently consumes messages from Kafka one at a time, and only after it returns the data back to Kafka does it start consuming another message.

My code:

app = faust.App('app_faust', broker=[...], store='rocksdb://')



class Company(faust.Record):
    company_id: str
    task_id: int


topic_c = app.topic(
    'topic_name',
    key_type=bytes,
    value_type=Company,
)
topic_p = app.topic(
    'another_topic_name',
    key_type=bytes,
    value_type=Company,
)

@app.agent('topic_name', sink=[topic_p], value_type=Company, concurrency=4)
async def test(data: faust.Stream):
    async for company in data:
        if data and company.company_id and company.task_id:
            yield some_function(company)

Any idea what am I doing wrong? I am using Faust 1.10.4

iam.mattevans
  • 51
  • 5

1 Answers1

1

according to documentation:

The agent modifying the table cannot process the source topic out of order, so only agents with concurrency=1 are allowed to update tables.

it works when your app has no rocksdb.

https://faust.readthedocs.io/en/latest/userguide/application.html?highlight=concurrency#app-table-define-a-new-table

Everton Arakaki
  • 59
  • 4