JOB to Dataflow using function

Question

Parameter google function

eu quero passar parameter para query

template file:

"steps":[
      {
         "kind":"ParallelRead",
         "name":"s1",
         "properties":{
            "bigquery_export_format":"FORMAT_AVRO",
            "bigquery_flatten_results":true,
            "bigquery_query":"select * from `myproject2497.teste.teste`",
            "bigquery_use_legacy_sql":false,
            "display_data":[
               {
                  "key":"source",
                  "label":"Read Source",
                  "namespace":"apache_beam.io.iobase.Read",
                  "shortValue":"BigQuerySource",
                  "type":"STRING",
                  "value":"apache_beam.io.gcp.bigquery.BigQuerySource"
               },
               {
                  "key":"query",
                  "label":"Query",
                  "namespace":"apache_beam.io.gcp.bigquery.BigQuerySource",
                  "type":"STRING",
                  "value":"select * from `myproject2497.teste.teste limit 1`"
               },

function google cloud: blob.upload_from_string(timestamp)

parameters = {"bql": bql}
jobname = "poc"    
gcsPath="gs://exemplebucket1321/teste/templates/Bee"
body = {
    "jobName": "{jobname}".format(jobname=jobname),
    "parameters": parameters
}

Would you mind editing to clarify the question a bit please, any of these would help. (1) Are you using the bql statement to determine the start date and passing that in as a parameter to the template in order to load a start date from BigQuery? (2) Where in your code are you saving the start date to use? (3) Or are you simply defining a query which determines which data you read in bql, which includes the start date. (4) What error are you seeing, when you say it cannot read the information? (5) Please provide the command line you ran, or parameters you used in the template UI, if possible. — Alex Amato, Mar 17 '20 at 20:33
guy in fact what I need now would be something like passing the parameter to query in the function — Alan Batista, Mar 18 '20 at 14:01

score 0 · Accepted Answer · answered Mar 18 '20 at 21:15

Please see the documentation on creating:

https://cloud.google.com/dataflow/docs/guides/templates/creating-templates

And RunningTemplates:

https://cloud.google.com/dataflow/docs/guides/templates/running-templates

Note that you will only be pass in parameters in the template code which is already using a ValueProvider (without changing the source). See the link below for the list of IOs with ValueProvider options. For BigQuery you will need Java SDK 2.0+

https://cloud.google.com/dataflow/docs/guides/templates/creating-templates#pipeline-io-and-runtime-parameters

Since you are using BigQuery as the source, I believe following the instructiosn in the above two links will resolve your issue

JOB to Dataflow using function

1 Answers1