Is data sink phase take a lot of time during flink job execution?

Asked May 02 '20 at 22:37

Active May 02 '20 at 22:37

Viewed 111 times

I'm using VM and my cluster consists of 3 taskmanagers and the master is jobmanager and taskmanager too (4 taskmangers and one job manager) about 12 gb ram and 6 processors,and using flink-1.7.2 version but after I run a flink job

./bin/flink run -p 4 -q examples/gelly/flink-gelly-examples_*.jar --algorithm PageRank --input CSV --type integer --simplify directed --input_filename WikiTalk.txt --input_field_delimiter $'\t' --output print,

I found that the job is still running on the datasink(collect)

asked May 02 '20 at 22:37

Soad Ahmed

It depends on the size of the data set you have. Have you tried it with a larger dataset? – damjad May 03 '20 at 17:07
@chuckskull Wikitalk dataset, its size 66.5 MB – Soad Ahmed May 04 '20 at 17:14
@chuckskull is the dataset so large to take more than one day – Soad Ahmed May 06 '20 at 20:01
It seems very weird. Le me run on my PC. – damjad May 06 '20 at 20:14
Where did you download WikiTalk file? – damjad May 06 '20 at 20:15
@chuckskull from here https://snap.stanford.edu/data/wiki-Talk.html – Soad Ahmed May 06 '20 at 20:45
It looks weird, but it could also be that the output size explodes (which shouldn't happen on PageRank). – Arvid Heise May 07 '20 at 07:26
@ArvidHeise , what did u mean about the output size exploding?, and what should I do in this case or my problem? – Soad Ahmed May 08 '20 at 19:16
In general for any kind of input dataset, the output could be much larger (depends on the algorithm and the inherent properties of the dataset). For example, consider an algorithm that returns the powerset of the input data. That's when the output "explodes". In that case, it's expected that output takes much longer. You can usually check that by using a sample of the input data. – Arvid Heise May 09 '20 at 16:42
@ArvidHeise, OK, I will try a sample and see the result, could u check my command line if it has something wrong? – Soad Ahmed May 09 '20 at 21:41
The trailing comma looks suspicious, but I have not worked with gelly yet. – Arvid Heise May 11 '20 at 10:12

Is data sink phase take a lot of time during flink job execution?

0 Answers0