Trying to figure out why I'm getting these errors. A quick search just resulted in answers that referred to a broken version, but it doesn't seem to be the case here. Creating the template works fine, but when I run it (and as I pass the limit arg) I get the error below. The idea is to build up the query based off of arguments provided in the template. If there's a better alternative to doing this, I'm open for it.
Code:
class Options(PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
parser.add_value_provider_argument(
'--limit',
default=0,
type=int,
help='Limit the amount of rows retrieved'
)
...
def from_bq(options):
with beam.Pipeline(options=options) as p:
(p
| 'Read From BQ' >> beam.io.ReadFromBigQuery(query=NestedValueProvider(options.limit, create_query), use_standard_sql=True)
)
def create_query(limit):
query = """
SELECT * FROM ...
"""
if limit > 0:
query = query + " LIMIT {limit}".format(limit=limit)
return query
Error:
raise error.RuntimeValueProviderError('%s not accessible' % obj)
apache_beam.error.RuntimeValueProviderError: NestedValueProvider(value: RuntimeValueProvider(option: limit, type: int, default_value: 0), translator: create_query) not accessible [while running 'Read From BQ/Read/Split-ptransform-324']
Running apache-beam version 2.27.0.
