Questions tagged [flink-batch]
158 questions
0
votes
0 answers
Is data sink phase take a lot of time during flink job execution?
I'm using VM and my cluster consists of 3 taskmanagers and the master is jobmanager and taskmanager too (4 taskmangers and one job manager) about 12 gb ram and 6 processors,and using flink-1.7.2 version but after I run a flink job
./bin/flink run…
Soad Ahmed
- 13
- 6
0
votes
1 answer
Flink read data from Hadoop and publish to Kafka
I have a requirement to read data from HDFS and publish it to a Kafka topic. Because they are part of DataSet and DataStream APIs, is it possible to do what I'm looking for in a single job?
Harshith Bolar
- 728
- 1
- 10
- 29
0
votes
0 answers
what that mean 'the type serializer factory could not load its parameters from the configuration'?
I have a problem when I run this command
./bin/flink run -p 4 -q examples/gelly/flink-gelly-examples_*.jar --algorithm PageRank --input CSV --type integer --simplify directed --input_filename WikiTalk.txt --input_field_delimiter $'\t' --output…
Soad Ahmed
- 13
- 6
0
votes
1 answer
Can anyway Dataset transformation-: "Distinct()" be used in Datastream in Flink?
I was wondering if in anyway Flink's datastream API be used to remove duplicates from records coming in (may be over a particular time window), just like in Dataset API which provides with a transformation called "Distinct". Or in anyway if dataset…
Anish Sarangi
- 172
- 1
- 14
0
votes
1 answer
AvroInputFormat returns set of object addresses instead of values
I'm writing some data using flink AvroOutputFormat,
val source: DataSet[Row] = environment.createInput(inputBuilder.finish)
val tableEnv: BatchTableEnvironment = new BatchTableEnvironment(environment, TableConfig.DEFAULT)
val table: Table =…
codebot
- 2,540
- 3
- 38
- 89
0
votes
0 answers
Is there an alternative for GroupReduceFunction running apache-flink java in parallel?
The code below is running locally but not on the cluster. It hangs on GroupReduceFunction and do not terminates even after hours (it takes for large data ~ 9 minutes to compute locally). The last message in the log:
GroupReduce (GroupReduce at…
anno2019
- 1
- 1
-1
votes
1 answer
Use-Defined Aggregate Function in Flink - No match found for function signature
i would like to keep all raw rows per key in "Select .. From .. GROUP BY .." query in Flink. I defined a AggregateFunction called RowToJsonAgg which aggregates rows into a Json string.
class RowToJsonAgg extends AggregateFunction[String,…
Grant
- 500
- 1
- 5
- 18
-1
votes
1 answer
How to specify the time for which the state should be maintained in Flink CEP
Let me explain a scenario which I need to handle. Let us assume three devices A,B,C are sending logs to the flink CEP for processing. Let us assume the pattern as A followed by 5 mins B which is followed by C in 5 mins. Let us assume a scenario…
JDForLife
- 91
- 2
- 10