1

I've create template for Dataflow which is reading messages from Pub/Sub The problem is that NestedValueProvider is not possible to modify sended via options value and I do not have clue why - below some screens

Passed parameter:

enter image description here

Code:

enter image description here

Result:

enter image description here

If anyone would ask why I needto nest it - I wanted to only require to add subscriptionID and just add prefix with rest of the path..

EDIT

My steps:

1.Created pipeline which is reading from Pub/Sub subscription and for subscription name is using NestedValueProvider for creating subscription name

pipeline.apply(
                            "Read PubSub messages",
                            PubsubIO.readStrings().fromSubscription(
                                    NestedValueProvider.of(
                                            options.getSubscriptionID(),
                                            input -> "projects/ozone-analytics-dev/subscriptions/" + input)
                            )
                    )

2.Created template

3.Running template with valid and existing subscriptionID

4.Seeing an error while creating dataflow job (I think that job didn't even start and this error is through some pre validation cause I can run this flow locally or normally with DataflowRunner but not via template) enter image description here

Kapitalny
  • 651
  • 2
  • 8
  • 17
  • Whole pipeline is working correctly, after running actually (through DirectRunner or by running DataflowRunner job not through template) Running via template is making some invalid validations which are preventing this even from running I think – Kapitalny May 26 '20 at 12:41
  • Can you try passing your subscription in this format? projects//subscriptions/ – rmesteves May 27 '20 at 08:19
  • For example, if your project id is 1234 and your subscription name is bbb, that would be projects/1234/subscriptions/bbb – rmesteves May 27 '20 at 08:20
  • Yes, this will work - but my problem is that I would like to wrap my subscriptionID like `bbb` with full name via NestedValueProvider (`projects/1234/subscriptions/bbb`). But when I start template (event if this wrapping actually works after run) Google do not want to even create working template. There is some kind of invalid validations occuring or sth. As I said I can run it via console on Dataflow - only template creation is performing something weird – Kapitalny May 27 '20 at 08:38
  • I understand that you want to use only the subscription name. Can you share the steps of what you did in which order? I would like to understand better what is happening – rmesteves May 27 '20 at 09:30
  • Ok, I've added at the end of my description - steps that I've made for this error to occur. I believe that there is some pre validation happening while running job from template which is ignoring NestedValueProvider somehow – Kapitalny May 28 '20 at 09:02
  • There is probably an interface limitation that expects that specific pattern for subscriptions. Have you tried running it through command line? – rmesteves May 28 '20 at 09:11
  • Yes as I've written running through command line is working fine - running with DirectRunner / DataflowRunner is fine. But running it through template via GCP it is causing this `Workflow failed` error I think that this definitely is some kind of bug – Kapitalny May 28 '20 at 09:26
  • So definitely this is an interface limitation. If you think this feature would be something nice to have, you could create a feature request and Google will conclude if its feasible or not – rmesteves May 28 '20 at 09:32
  • Yeah I'll create some ticket, thanks for your time on my problem! – Kapitalny May 28 '20 at 12:14
  • You're welcome :) Feel free to summarize the discussion in an answer or let me know if I can do that – rmesteves May 28 '20 at 12:40

1 Answers1

1

After analyse - this is rather Google Dataflow bug which only is related to template creation validation (because every other way of Dataflow run is behaving correctly)

I've created issue for that https://issuetracker.google.com/u/1/issues/157584222

And for now it is the answer for that problem

Kapitalny
  • 651
  • 2
  • 8
  • 17
  • 1
    I wish with all my heart I'd found this days ago and not wasted about 3 days pulling my hair out. Thank you for posting this though, you've saved me from going completely insane – blablabla Apr 20 '21 at 15:47