0

I would like to create a Stackdriver dashboard to monitor the number of elements being Read/Wrote by my Pipelines. The dataflow/job/element_count metric seems to cover this use case, unfortunately I can't get it to work properly (cf picture)

stackdriver dashboard

Did anyone have this problem before? Would you know how to filter this metric in order to have only the element count for the Read/Write PTransforms?

Thanks !

Kevin Maillet
  • 1
  • 1
  • Doesn't the `dataflow/job/element_count` metric explained [in step 5 here](https://cloud.google.com/dataflow/pipelines/stackdriver-monitoring) do what you're looking for? – Mitch Lillie Apr 24 '17 at 15:52
  • Thank you for your answer, my question was terribly vague and inaccurate. I just edited it ! Thanks – Kevin Maillet Apr 26 '17 at 08:09

1 Answers1

1

You should be able to just create a dashboard by picking Resource Type as 'Dataflow Job' and Metric Type as 'element count'.

As long as your source and sink are just reading and writing the amount of elements, you should be able to use the element counts on the output collections. You can put them on the graph and mouse over to see the separate amounts. I am not sure of a way to show only one though if that is what you want to do.

Lara Schmidt
  • 309
  • 2
  • 6
  • Thank you for your answer Lara. As you can see on my updated question, the `dataflow/job/element_count` doesn't give me any data for a large number of my pipelines. I don't understand this since I use basic Textio sink and source. Would you have any idea of what might be causing this behaviour? Thanks ! – Kevin Maillet Apr 26 '17 at 08:17
  • Ah, that is very odd. It should be populating that field for all pipelines. Do you have a dataflow job id and I can take a look. Sorry for the slow response, new to StackOverflow and missed the reply. :) – Lara Schmidt Apr 26 '17 at 18:24
  • Do you see the element count in the Dataflow UI? If you do, it should be in stack driver. Default charts only look back a really short amount of time. So it might just be that you need to increase what period your chart is covering. – Lara Schmidt Apr 26 '17 at 18:30
  • No worries, just starting with stackoverflow myself ;) Here is a job id "2017-04-26_03_07_58-4399039998964834055". In the dataflow UI I can see the number of element for my Read/Write PTransforms. In the stackdriver dashboard I did try to set the time period to 1 week and 1 month, I still couldn't see the element count for a large portion of my pipelines ... Thank you for your help, I appreciate it – Kevin Maillet Apr 27 '17 at 13:43
  • So I'm not seeing anything out of place on the Dataflow side. I sent a note to the stackdriver team. This is likely some kind of bug or maybe something on your end is mis-configured. In the mean time, you can try using the metric explorer instead and seeing if you have more luck with that. Or filing an issue yourself may help as well since then you can have more direct contact. Sorry I can't be more helpful! – Lara Schmidt Apr 27 '17 at 17:29
  • Thanks Lara. For some reason I can't explain, my stackdriver dashboard now renders properly the element count metric. I didn't do any modification from my side so this is very mysterious. Anyhow it's working now. Thank you very much for your time – Kevin Maillet Apr 28 '17 at 07:06
  • I just realised that only pipelines launched from templates don't appear in the dashboard. Is there any known issues about that? – Kevin Maillet Apr 28 '17 at 07:12
  • I sent a note to stackdriver to take a look at pipelines running under templates to see if there is any issue. I also noticed that pipeline has an odd name. Maybe you could try running with a simpler name and see if that fixes it? Possibly some parser error (going out on a limb now :) ) – Lara Schmidt Apr 28 '17 at 21:38