Stack Overflow
Stack Overflow

Questions tagged [faust]

Faust is a stream processing library, porting the ideas from Kafka Streams to Python.

Faust is used to build high performance distributed systems and real-time data pipelines that process billions of events every day.

Faust provides both stream processing and event processing, sharing similarity with tools such as Kafka Streams, Apache Spark/Storm/Samza/Flink,

It does not use a DSL, it’s just Python! This means you can use all your favorite Python libraries when stream processing: NumPy, PyTorch, Pandas, NLTK, Django, Flask, SQLAlchemy, ++

Faust requires Python 3.6 or later for the new async/await syntax, and variable type annotations.

Repository: https://github.com/robinhood/faust

96 questions
1
vote
0 answers

Faust. How to create a compacting topic

I want to create a topic that is automatically compacted by kafka, using faust stream. I use code like: import asyncio import random import faust app = faust.App( 'mysample', broker=<...>, value_serializer='raw', …
Роман Коптев
  • 1,555
  • 1
  • 13
  • 35
1
vote
1 answer

Two agents with different filters on one kafka topic. Acknowledgment in Faust Stream

I want to have two faust agents listening the same kafka topic, but every agent uses its own filter before process the events, and their event sets doesn't intersect. In the documentation we have an…
Роман Коптев
  • 1,555
  • 1
  • 13
  • 35
1
vote
1 answer

make faust agent wait for other agents to finish

Is there any way to make faust agent run when some other agents finish? For example Agent1 + | | | Agent2 +------------->Agent_final | | | + Agent3 Agent_final should start when…
palkan
  • 321
  • 2
  • 5
1
vote
0 answers

Why does my apache kafka consumer randomly ignores queued messages?

This is probably an eisenbug so I'm not expecting hard answers but more hints on what to look for to be able to replicate the bug. I have an event-driven, kafka-based system composed of several services. For now, they are organized in linear…
Chobeat
  • 3,445
  • 6
  • 41
  • 59
1
vote
1 answer

Autodiscover python decorators

I was wondering if there is a standardized approach or best practice to scan/ autodiscover decorators like it is done here but also in several other libs like Django, Flask. Usually a decorator provides extra/ wrapped functionality right at the time…
trbck
  • 5,187
  • 6
  • 26
  • 29
0
votes
1 answer

Want to use Faust with Event Hub kafka

We are migrating from kafka to eventhub and as eventhub itself uses kafka in background, we are trying to use our existing code using python "faust" with some connection changes to connect to eventhub, but it is somehow not working.(we don't want to…
jaspreet sethi
  • 1
  • 1
0
votes
0 answers

How to process empty list of messages in faust?

Context I want to add heartbeat monitoring mechanism to my worker-consumer-processes. Heartbeat means that either some useful work was done, or processes attempted to fetch data but received nothing from broker. It will be implemented with…
aiven
  • 3,775
  • 3
  • 27
  • 52
0
votes
0 answers

Rocks DB benefits when using Faust

According to faust-streaming it is recommended to use Rocks DB for the fast recovery of the faust tables. Say there is one machine running a consumer with the faust table and Rocks DB. Rocks DB manages in-memory state of the table backed by the…
Plesiosaurus
  • 1
  • 1
0
votes
0 answers

Faust deserializing model when model field is another model

I have a model that is a little complicated where one field can be one of several other models which looks similar to the following: '''These are the options''' class ChatterFilter(Record, serializer='json'): counts:int time:int …
Fonty
  • 239
  • 2
  • 11
0
votes
0 answers

Failing to set up a log formatter class instance using Faust

Im trying to add a class of log formatter to the faust logging_config but getting an error with not mush exlained: CANNOT SETUP LOGGING: ValueError("Unable to configure formatter 'exp_formatter'") from File "det_to_events/main.py", line 246, in…
MC LinkTimeError
  • 67
  • 10
0
votes
1 answer

Logs are printed twice using faust

Using python Faust, all of my logs are doubled. As loggers are hierarchal, i want to prevent the downstream logger father.son from logging anything and propagate all the logs to upstream father logger. I tried: 1 - setting disable_existing_loggers…
MC LinkTimeError
  • 67
  • 10
0
votes
1 answer

How to start multiple faust app in the same time?

I'm new user of Faust and don't know how to fix the problem when I ran 3 faust apps in the same time. Specifically: I have 3 python file, In each, I run 1 service for listening from kafka server. Each file contains code as below, the different in…
ShanN
  • 831
  • 1
  • 9
  • 20
0
votes
0 answers

how is it used to deserialize JSON messages from Kafka?

import faust app = faust.App('demo-streaming', broker='localhost:9092') class Greeting(faust.Record, serializer='json'): source_id: str source_name: str author: str title: str description: str url: str urlToImage: str …
Mario
  • 21
  • 3
0
votes
2 answers

Sending messages to Faust topic from local file / List

I want to be able to either consume messages from a Kafka broker or a local file with data in it. How do I do this with Faust without writing a very similar function without Faust that just uses a simple for loop to iterate over messages? Or is it…
clueless_robot
  • 3
  • 2
0
votes
1 answer

Can't Access configparser environment variables from env.ini file in faust (kafka streaming)

My Project is on FastAPI and the structure goes like this. - project - app - kafka_layer - faustworker.py - core - configs.py - env.ini My env.ini file goes like this [DEFAULT] DATABASE_URL=url_to_db I've tried…
Raheel Siddiqui
  • 95
  • 6
1 2 3
4
5 6 7