Stack Overflow
Stack Overflow

Questions tagged [faust]

Faust is a stream processing library, porting the ideas from Kafka Streams to Python.

Faust is used to build high performance distributed systems and real-time data pipelines that process billions of events every day.

Faust provides both stream processing and event processing, sharing similarity with tools such as Kafka Streams, Apache Spark/Storm/Samza/Flink,

It does not use a DSL, it’s just Python! This means you can use all your favorite Python libraries when stream processing: NumPy, PyTorch, Pandas, NLTK, Django, Flask, SQLAlchemy, ++

Faust requires Python 3.6 or later for the new async/await syntax, and variable type annotations.

Repository: https://github.com/robinhood/faust

96 questions
0
votes
1 answer

Is there any way to define app.Table without using Record in faust?

I'm currently using schema registry and faust to process stream data. The reason I try to avoid using faust.Record is the schema can be dynamically changed and I don't like to change the code(class inheriting faust.Record) every time it happend. But…
user3595632
  • 5,380
  • 10
  • 55
  • 111
0
votes
0 answers

Launching Faust consumer as a separate thread

I have a platform where I want to launch as a separate thread a Faust consumer. Anyway, from the documentation I always find asynchronous implementation where the consumer is running on a process, but there is no mention about a solution for…
Andrea Fresa
  • 351
  • 2
  • 18
0
votes
0 answers

How to make audiokit polyphonic node using STK physical models?

I have been looking for a way to convert STK physical models to audiokit polyphonic node. There are examples inside Audiokit repo, but complete process of triggering the notes are getting complex. If someone provides the easy way to integrate STK…
The YooGle
  • 81
  • 2
  • 2
0
votes
1 answer

Create polyphonic dsp node in audiokit 5

I am searching for the resources to create a polyphonic dsp node in audiokit 5, so that I can connect and use it with AudioEngine. For c++ dsp, I am using faust. AudioKit with faust single voice node is working for me by using faust2audiokit…
The YooGle
  • 81
  • 2
  • 2
0
votes
1 answer

Read a list of JSON from kafka with faust

Each record in Kafka is a list of dictionaries (serialized using lambda x: json.dumps(x).encode('utf-8') and I'm trying to use Faust. I understand that I can pass the schema by creating a class like so: class row(faust.Record): ID: int …
Mohamed Yasser
  • 641
  • 7
  • 17
0
votes
0 answers

Faust streaming app - Avro schema not published to Kafka cluster

I have set up a Kafka cluster with a schema-registry using docker containers. Then I built a Faust streaming app that sends messages to a Kafka topic. The messages are sent to the Kafka topic and stored as expected. But I cannot see the defined Avro…
CarloP
  • 99
  • 1
  • 12
0
votes
1 answer

Faust streaming agent slow to 'wake up' after idle period

I have a pipeline where a message is produced to topic A, which is processed by a stream processor and sends enriched data to topic B. Topic B is consumed by 3 other stream processors which independently perform a small part of the calculation (to…
Fonty
  • 239
  • 2
  • 11
0
votes
1 answer

Accessing raw faust/kafka message while using take method

I have a need to batch the events in a Faust stream, so I'm using the take() method. However, I would also like to access the header message, particularly a timestamp. Ordinarily you would access the header using: async for event in…
Fonty
  • 239
  • 2
  • 11
0
votes
1 answer

How to specify Faust arguments when using app.main()

Currently I run several workers using the command line with the following syntax: faust -A :app -D worker -l info -p 6066 and it works fine. But it seems it would be much nicer to run the .py file and have the log-level,…
Fonty
  • 239
  • 2
  • 11
0
votes
2 answers

await Faust Agent ask() never receive from yield generator

Hi I am trying to integrate faust with fastapi endpoints following this example: toh995/fastapi-faust-example I am working with a simple DummyOrder model class DummyOrder(faust.Record,MyAvroModel,serializer='avro_order_codec'): order_id: str …
Gyslutz7749 Gyslutz7749
  • 1
0
votes
1 answer

Faust error: Cannot modify table key from outside of stream iteration

I've implemented a Kafka topic that tries to buffer multiple events for efficiency but when I try and modify a Faust table I get the error message Cannot modify table key from outside of stream iteration class BroadcastCounts(faust.Record): …
chrislondon
  • 12,487
  • 5
  • 26
  • 65
0
votes
1 answer

"encoding without a string argument" error in Faust agent that sends data to a sink?

I'm trying to send some data from one Kafka topic to another, using sink in Faust agent, and I keep receiving this error: File "/.../venv/lib/python3.8/site-packages/schema_registry/serializers/message_serializer.py", line 49, in return…
gorqiwi
  • 23
  • 4
0
votes
3 answers

How to send none values (tombstones) to a kafka topic using Faust

I'm trying to send data from one topic in Kafka to another using Faust. If the value in the original topic is None (the message is a tombstone), I'm sending the current key with a None value to the target topic. async def order_delete(key,…
Dmitriy Fedotov
  • 1
0
votes
0 answers

How can i send large message to kafka producer using python?

if I send the largest Json to the Kafka server it will show this kind of error, How can I increase message.max.bytes=15728640 and replica.fetch.max.bytes=15728640 in Kafka. I tried to increase byte level as below it won't work The send buffer…
Ravi Devarasu
  • 1
  • 1
0
votes
1 answer

Faust doesn't like relative path

I'm trying to clean up my code and have moved the models.py file to the top level, as other modules other than the faust ones will use this now. The folder structure is below (albeit cut down for simplicity) App | ├── models | ├── models.py | ├──…
Fonty
  • 239
  • 2
  • 11
1 2 3 4
5
6 7