Questions tagged [event-stream-processing]
43 questions
0
votes
1 answer
Is it possible to achieve Exacly Once Semantics using a BASE-fashioned database?
In Stream Processing applications (f. e. based on Apache Flink or Apache Spark Streaming) it is sometimes necessary to process data exactly once.
In the database world something equal be achieved by using databases that follow the ACID criteria…
MW.
- 544
- 5
- 19
0
votes
1 answer
Issues connecting Snowplow with Kafka on Confluent Cloud
I have managed to get an end to end local implementation working within Docker using Web Tracker, Scala Stream Kafka Collector, Stream Enrich, Druid and Metabase.
Before building a working staging environment within Kubernetes using Helm, I wanted…
Steven
- 476
- 3
- 7
0
votes
1 answer
In-order processing in Azure event hubs with Partitions and multiple "event processor" clients
I plan to utilize all 32 partitions in Azure event hubs.
Requirement: "Ordered" processing per partition is critical..
Question: If I increase the TU's (Throughput Units) to max available of 20 across all 32 partitions, I get 40 MB of egress. Let's…
teeboy
- 408
- 3
- 13
0
votes
0 answers
PHP closing connections when too many
recently, I decided to load test my website on some page that require server sent events (event streams) which means that PHP needs to handle many opened connections (with scripts on loop). When approaching the ~140 opened connection, the…
i thirzty
- 51
- 1
- 1
- 4
0
votes
1 answer
Using provided topics for changelog and repartition while aggregating in Kafka Stream Processing
I am using Kafka stream processing for aggregating data from source object with Springboot.
@Bean
public java.util.function.Consumer> processSourceObject() {
Serde SourceObjectSerde = new…
ADCDER
- 61
- 1
- 4
0
votes
1 answer
Keep only alphanumeric characters in string data
I'm getting string data in a stream and I would like to keep only the alphanumeric characters. I noticed that Siddhi offers a regexp function, as mentioned here. But the thing is it returns a Boolean instead of the modified string. Is there anyway…
dilip sundar
- 91
- 1
- 7
0
votes
1 answer
Ordering event with Esper java
I'm developing IoT Platform and what i need to do is preserve the events order.
For example, Sometimes slow network may causes the problem that previous events reached after newer events. In this case, I need to reorder the events with timestamp…
Sehun Park
- 113
- 1
- 2
- 11
0
votes
1 answer
When trying to execute an Siddhi APP using an event-stream generated by JMeter the RAM usage gets out of control
When trying to simulate an event-stream with JMeter and use it as source on siddhi it works for a little time but ends with RAM being overused and the execution of the program stops.
I tried executing the code with a database,without a database,…
SereniFXX
- 1
- 1
0
votes
1 answer
How to create rolling windows with Apache Beam? Not sliding or fixed but a rolling window
Say I want to calculate the average of certain metric over the last 10 mins, after each minute and compare it to the average of the same metric over the last 20 mins, after each minute. I need 2 windows (Not 10 Sliding windows vs 20 Sliding windows)…
Nishant
- 395
- 4
- 6
0
votes
1 answer
Use Kinesis Analytics for analyzing events and related missing events, separated in time?
I have a stream of events for various devices that can either be "connected" or "disconnected".
I.e. an event has the following structure:
timestamp
device_id
event ("connected" or "disconnected")
I want to trigger an action instantly when a…
user373481
- 53
- 3
0
votes
1 answer
Flink sql Query on DataStream (Apache Flink Java)
I am totally noob to Apache flink. Just trying to get my hands dirty. I have the following scenario.
Datastream for Events
Datastream for Events
Datastream for rules
Combined these two datastreams based on ruleID
Now I have a datastream of tuple3…
0
votes
2 answers
Domain data / reference data implementation?
Fairly simple question, but I have not been able to find a way to do it via the docs.
I am interested in a core set of reference data that all events can be compared against. In the simple example, there are segmentation contexts that talk about…
drmcl
- 1
- 3
-1
votes
2 answers
Problem in reading event stream in AWS Lambda. Nodejs code working locally as desired but not in AWS Lambda
Here's the workflow:
Get a https link --> write to filesystem --> read from filesystem --> Get the sha256 hash.
It works all good on my local machine running node 10.15.3 But when i initiate a lambda function on AWS, the output is null. Some problem…
damn
- 1
- 1