Stack Overflow
Stack Overflow

Questions tagged [faust]

Faust is a stream processing library, porting the ideas from Kafka Streams to Python.

Faust is used to build high performance distributed systems and real-time data pipelines that process billions of events every day.

Faust provides both stream processing and event processing, sharing similarity with tools such as Kafka Streams, Apache Spark/Storm/Samza/Flink,

It does not use a DSL, it’s just Python! This means you can use all your favorite Python libraries when stream processing: NumPy, PyTorch, Pandas, NLTK, Django, Flask, SQLAlchemy, ++

Faust requires Python 3.6 or later for the new async/await syntax, and variable type annotations.

Repository: https://github.com/robinhood/faust

96 questions
2
votes
0 answers

Faust agent hangs without any error. Can't figure out what is the issue

I am using faust and asyncio to process mt kafka stream. I am also using aiomysql to create connection pool. When I start the worker, it process the messages from the stream but hangs after processing some of them. There are no error messages in the…
Rahul Hindocha
  • 170
  • 2
  • 14
2
votes
0 answers

faust Record does not deserialize avro Union type correctly

I'm using faust-streaming and python-schema-registry-client to serialize faust.Record classes to avro. When I try to deserialize a Union of two complex types however faust cannot reconstruct the correct records and only presents a dictionary instead…
bbh
  • 23
  • 2
2
votes
1 answer

Tumbling window 'expires' parameter less than 'size' doesn't work

Steps to reproduce I'm trying to aggregate some data in the tumbling window then apply the process function to the data in the window. I'm using the expires argument to handle late events (let's imagine we can get an event that belongs to the n…
Dmytro Shulikov
  • 31
  • 4
2
votes
3 answers

Using python compiled protobuf pb2 as key and value serializer

I am trying to read data from a kafka topiv which has been serialized using google's protobuf. I compiled the proto files using protoc which generated pb2 files. Now i am trying to use faust and create a stream processor but i can't find the correct…
Yuv Joodhisty
  • 71
  • 1
  • 4
2
votes
0 answers

Trying to create a cronjob in django using faust

I'm trying to create a cronjob in django using faust. If I create a simple cronjob printing somethin' on screen it works, but if I try to use some ORM thing it doesn't. @sync_to_async def get_products(): return…
Yuri Costa
  • 199
  • 9
2
votes
0 answers

How to get faust to send raw replies

I'm trying to get replies like bytes using Faust Stream. But it always encode replies as sting (because faust uses json codec for replies). Is it possible to get replies just as bytes? import asyncio import faust app = faust.App( …
Роман Коптев
  • 1,555
  • 1
  • 13
  • 35
2
votes
1 answer

How do I use concurrency in faust?

I'm working with faust and would like to leverage concurrency feature. The example listed doesn't quite demonstrate the use of concurrency. What I would like to do is, read from kafka producer and unnest json. Then the shipments are sent to a…
user3327034
  • 395
  • 3
  • 13
2
votes
1 answer

How to send data to a Websocket from a faust application

I am working on an usecase at the moment using Kafka and robinhood's faust to process the data coming from Kafka. I have succeeded doing the computation and the results I need are being printed to the console my faust worker is running in. Now I…
LukasM
  • 43
  • 8
2
votes
1 answer

How to check if a new record has been sent in a given time period using kafka and faust

I am using a test setup including confluent platform (docker) and am processing records with the following information: Sensor-ID, timestamp, value. Using robinhood's faust (similar to Kafka Streams but in python) I am trying to do the…
LukasM
  • 43
  • 8
1
vote
0 answers

Agents of Faust streaming stop event consumption after some days

I have an application using faust-streaming. The application has around 20 agents each consuming events from different Kafka topics. The problem is, after several day, the agents stop consuming and I have to restart application container. Here is an…
Mr Alihoseiny
  • 1,202
  • 14
  • 24
1
vote
1 answer

Ensure faust consumer/agent seeks to offset 0 after rebalance

Not much point explaining why this needs to happen, but we need to have a worker always read from the beginning of a topic on startup and after a rebalance. The following snippet does work on start to ensure that when you start a worker it retrieves…
Fonty
  • 239
  • 2
  • 11
1
vote
1 answer

Why is faust stream not printing any message?

My faust set up is not printing any messages to console. I originally thought this was an authorization issue, but I've been able retrieve raw messages following the example here (How to get faust to send raw replies). I want to be able to retrieve…
Sam Comber
  • 1,237
  • 1
  • 15
  • 35
1
vote
1 answer

Setting up CORS with Faust webview

I'm attempting to directly post information to a Kaka topic via a Faust web view as per their example. However, I'm getting the familiar CORS issue in the response. Looking into the docs only leads me to see that I need to use aiohttp-cors to set up…
Fonty
  • 239
  • 2
  • 11
1
vote
1 answer

Is faust[rocksdb] a mandatory extension to be installed?

I am working on microservices architecture and using kafka to communicate b/n microservices. I am using python faust to implement producers and consumers. Is it mandatory to install faust[rocksdb] extension? When I install it, fails with a fatal…
Ranga
  • 113
  • 3
  • 9
1
vote
1 answer

How to run Faust from Docker - ERROR: Failed building wheel for python-rocksdb

I'm trying to run Python Faust from Docker. Based on this documentation: https://faust.readthedocs.io/en/latest/userguide/installation.html I created a simple Docker file: FROM python:3 ADD ./app/app.py / RUN pip3 install --upgrade pip RUN pip…
QbS
  • 425
  • 1
  • 4
  • 17
1
2
3 4 5 6 7