getting numOfRecordsIn using counters in Flink

Question

I want to show numRecordsIn for an operator in Flink and for doing this I have been following ppt by data artisans at here. code for the counter is given below

public static   class mapper extends  RichMapFunction<String,String>{

        public   Counter counter;

        @Override
    public void open(Configuration parameters) throws Exception {
        super.open(parameters);

        this.counter = getRuntimeContext()
                .getMetricGroup()
                .counter("numRecordsIn");
    }


    @Override
    public String map(String s) throws Exception {
        counter.inc();
        System.out.println("counter val " + counter.toString());
        return null;
    }
}

The problem is that how do I specify which operator I want to show number_of_Records_In?

score 2 · Accepted Answer · answered Oct 23 '17 at 12:23

2

Metric counter are exposed via Flink's metric system. In order to take a look at them, you have to configure a metric reporter. A description how to register a metric reporter can be found here.

answered Oct 23 '17 at 12:23

Till Rohrmann

13,148
1
25
51

score 2 · Answer 2 · answered Oct 23 '17 at 18:19

Flink includes a number of built-in metrics, including numRecordsIn. So if that's what you want to measure, there's no need to write any code to implement that particular measurement. Similarly for numRecordsInPerSecond, and a host of others.

The code you asked about causes the numRecordsIn counter to be incremented for the operator in which the metric is being used.

A good way to better understand the metrics system is to bring up a simple streaming job and look at the metrics in Flink's web ui. I also found it really helpful to query the monitoring REST api while a job was running.

getting numOfRecordsIn using counters in Flink

2 Answers2