Stack Overflow
Stack Overflow

Questions tagged [flink-cep]

FlinkCEP - complex event processing for Apache Flink

368 questions
1
vote
1 answer

How to coordinate speed between streams in Flink?

I have a raw log stream and an algorithm stream like below. As shown above, because the algorithm has windows, it takes a long time to produce results.The raw log stream run too fast, resulting in the output of the algorithm stream being discarded…
Brutal_JL
  • 2,839
  • 2
  • 21
  • 27
1
vote
1 answer

How to minimize Flink checkpoint state size

I have the following CEP PatternStream where the DataStream is partitioned based on entity ID because I am only interested in a pattern match if the entities have the same entity ID: PatternStream patternStream =…
sshum
  • 58
  • 6
1
vote
1 answer

read string datastream in Flink from socket without using netcat server

I have a case scenario in which I have a stream generator client which is generating multiple streams, merging them and sending it to socket and I want Flink program to listen to it as the server. As we know that server has to be turned up first, so…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
1
vote
1 answer

Designs for counting occurrences of events in streaming processing?

The following discussion is in the context of Apache Flink: Imagine that we have a keyedStream whose key is its id and event time is its timestamp, if we want to calculate how many events arrived within 10 minutes for each event. The problems need…
Leyla Lee
  • 466
  • 5
  • 19
1
vote
1 answer

merging datastreams of two different types in Flink or any other system

I want to use Flink for a remote patient monitoring case scenario which includes various sensors like gyroscope, accelerometer, ECG Stream, HR rate stream, RR rate etc. So in this case scenario it's not possible that we should have the same data…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
1
vote
1 answer

Apache Flink: How to count the total number of events in a DataStream

I have two raw streams and I am joining those streams and then I want to count what is the total number of events that have been joined and how much events have not. I am doing this by using map on joinedEventDataStream as shown…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
1
vote
1 answer

How to generate more than 1000 events / second in java using sleep time

I have a generator which generates events for Flink CEP, code for which is given below. Basically, I am using Thread.sleep() and I have read somewhere that java can't sleep less than 1 millisecond even we use System.nanoTime(). Code for the…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
1
vote
1 answer

Using Flink window and fold function, element missing?

When I try to aggregate elements using window and fold function, some of the elements are missed from getting aggregated. Consuming elements from Kafka (value:0, value:1, value:2, value:3) and aggregating them as odd and even values. Output is:…
Sharath
  • 407
  • 5
  • 16
1
vote
1 answer

Stateful Complex event processing with apache flink

I want to detect if two events happen in a defined timeframe based on two events that have the same identifier. For example a DoorEvent looks like this: 1 open …
Gert Kommer
  • 1,163
  • 2
  • 22
  • 49
1
vote
2 answers

adding Cassandra as sink in Flink error : All host(s) tried for query failed

I was following up with an example at https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/connectors/cassandra.html to connect Cassandra as sink in Flink My code for is shown below public class writeToCassandra { private static…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
1
vote
1 answer

Processing multiple patterns in Flink CEP in Parallel

I have following case scenario There are 2 Virtual machines which are sending streams to Kafka which are being received by CEP engine where warnings are generated when particular conditions are satisfied on the individual Stream. Currently, CEP is…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
1
vote
2 answers

Reading latest data from Kafka broker in Apache Flink

I want to receive the latest data from Kafka to Flink Program, but Flink is reading the historical data. I have set auto.offset.reset to latest as shown below, but it did not work properties.setProperty("auto.offset.reset", "latest"); Flink…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
1
vote
2 answers

Sending DataStream in Flink using sockets; serialization issue

I want to send Stream of data from VM to host machine and I am using method writeToSocket() as shown below: joinedStreamEventDataStream.writeToSocket("192.168.1.10", 6998) ; Here joinedStreamEventDataStream is of type…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
1
vote
2 answers

Write data from custom source to flink in continuous way

It is the first time I am working with Apache Flink (1.3.1) and have a question. In more detail I am working with flink-core, flink-cep and flink-streaming library. My application is an Akka ActorSystem what consumes messages from RabbitMQ and…
K.E.
  • 818
  • 2
  • 12
  • 28
1
vote
1 answer

Dynamic Stream SQL for Apache Flink CEP

I want to put stream SQL in Kafka to be consumed by Flink for CEP. Is this a good way ? I know that dynamic pattern definition is not allowed on Flink and i need to apply rules that they can change on unbounded event stream. To give an…
ccobanoglu
  • 192
  • 3
  • 13
1 2 3
…
24 25