Stack Overflow
Stack Overflow

Questions tagged [faust]

Faust is a stream processing library, porting the ideas from Kafka Streams to Python.

Faust is used to build high performance distributed systems and real-time data pipelines that process billions of events every day.

Faust provides both stream processing and event processing, sharing similarity with tools such as Kafka Streams, Apache Spark/Storm/Samza/Flink,

It does not use a DSL, it’s just Python! This means you can use all your favorite Python libraries when stream processing: NumPy, PyTorch, Pandas, NLTK, Django, Flask, SQLAlchemy, ++

Faust requires Python 3.6 or later for the new async/await syntax, and variable type annotations.

Repository: https://github.com/robinhood/faust

96 questions
1
vote
1 answer

Faust - How to visualize data inside tables?

I understood how to insert data inside the tables, but then how can i access it? How can i print on screen the data inside 'order_count'?
SimAzz
  • 138
  • 2
  • 14
1
vote
0 answers

Ensure that messages keep consistent order between topics

I have a faust application that has two topics. The first one receives raw data in the format: # Input of first topic { timestamp: 2021-08-24 05:35:24, value: 40, device_id: ABC } An async function interpolate consumes this topic and…
tschmelz
  • 480
  • 1
  • 4
  • 10
1
vote
0 answers

Faust-Streaming Crashes with error "Partition is not assigned"

We have recently switched to faust-streaming(0.6.9) from faust 1.10.4. Post this we have seen the applications crashing with the below exception. The application has multiple layers with aggregation and filtering of data at each stage. At each…
user2611300
  • 133
  • 2
  • 9
1
vote
1 answer

How to use concurrency in Faust?

I am trying to implement Faust library into my code to consume data from Kafka, run some code asynchronously and return data back to Kafka. I followed the documentation but seem to have made a mistake in my code (or in my understanding of how Faust…
iam.mattevans
  • 51
  • 5
1
vote
0 answers

Faust : Not seeing Messages Emitted from Tumbling Window when messages sent to different partitions

I created a test application to understand the window on-close callback functionality of the tumbling table. But I ran into an issue when running this application. In my test application, I have two timer tasks with each sending message with…
user2611300
  • 133
  • 2
  • 9
1
vote
1 answer

Batching Kafka Events using Faust

I have a Kafka topic we will call ingest that receives an entry every x seconds. I have a process that I want to run on this data but it has to be run on 100 events at a time. Thus, I want to batch the entries together and send them to a new topic…
Derek Johnson
  • 11
  • 2
1
vote
0 answers

Faust Application Seeing messages dropped in Tumbling Window

I am running a faust application that receives protobuf messages over a Kafka topic and inserts them into a tumbling window table with the user id as the key and value as the faust record containing the counts, user Id, and logged time( in seconds).…
user2611300
  • 133
  • 2
  • 9
1
vote
1 answer

Python Faust Tumbling Window not working when using relative_to_field

When the tumbling window used along with relative_to_field it is not working as expected. The values updated against a key itself are not getting reflected in the same window. It always returns the default value instead of the updated value. When I…
Dev Loper
  • 209
  • 1
  • 4
  • 18
1
vote
1 answer

understanding joining a faust topic with a faust table and partitioning

I have two topics: 1 topic with event data (EventData, let's say 5 partitions) -- logs on this topic use CustomerID as a key. 1 compact topic with enrichment data (EnrichmentKVs, let's say 3 partitions) -- logs on this topic use the same CustomerID…
booleys1012
  • 671
  • 3
  • 9
1
vote
1 answer

Unable to send message to faust agent/topic using webview when agents are built programmatically

I am trying to build agents programmatically and have been following this example. And I could see topics and agents when I look at faust -A <> agents that have been generated at the start of the script. The agent name is something like…
NRK
  • 21
  • 2
1
vote
1 answer

Proper way to resend messages to topic

I load messages from kafka topic to database. Loading to database can fail. Also I do not want to lose unsent messages. App code: import faust app = faust.App('App', broker='kafka://localhost:9092') source_topic =…
Альберт Александров
  • 807
  • 1
  • 15
  • 27
1
vote
1 answer

Future inside future always pending

P.S. Started an issue https://github.com/robinhood/faust/issues/702 Developing Faust-app: from concurrent.futures import ProcessPoolExecutor, as_completed import faust app = faust.App('my-app-name', broker='kafka://localhost:9092') sink =…
Альберт Александров
  • 807
  • 1
  • 15
  • 27
1
vote
0 answers

Make faust.Record created from dataclass works with optional fields

Currently i'am writting a python test so I can assure my make_faust_record() method works. This is how is looks like: def test_order_supports_optional_fields_records(): klass = make_faust_record(Order) payload = {"pos": {"hub": {"name":…
Vitor Mascia
  • 109
  • 2
  • 9
1
vote
2 answers

Running Faust Agent Synchronously

Check the below code @app.agent() async def process(stream): async for value in stream.take(5000, within=5): process(value) The agent takes 5000 records within 5 seconds asynchronously and process them. I don't want the agent to pick…
Rohit
  • 761
  • 1
  • 6
  • 8
1
vote
1 answer

Deserializing json object to Faust record where json has hyphenations in key names

I have a JSON object that looks like { "timestamp": "2020-06-24T15:32:56.775518Z", "record-type": "data", "operation": "load", "partition-key-type": "primary-key", "schema-name": "legacy", "table-name": "test" } and I'm trying to…
qbzenker
  • 4,412
  • 3
  • 16
  • 23
1 2
3
4 5 6 7