how to iterate over a list of values returning from ops to jobs in dagster

Question

I am new to the dagster world and working on ops and jobs concepts. \

my requirement is to read a list of data from config_schema and pass it to @op function and return the same list to jobs. \

The code is show as below

@op(config_schema={"table_name":list})
def read_tableNames(context):
    lst=context.op_config['table_name']
    return lst

@job
def write_db():
    tableNames_frozenList=read_tableNames()
    print(f'-------------->',type(tableNames_frozenList))
    print(f'-------------->{tableNames_frozenList}')

when it accepts the list in @op function, it is showing as a frozenlist type but when i tried to return to jobs it conver it into <class 'dagster._core.definitions.composition.InvokedNodeOutputHandle'> data type

My requirement is to fetch the list of data and iterate over the list and perform some operatiosn on individual data of a list using @ops

Please help to understand this
Thanks in advance !!!

score 0 · Answer 1 · answered Jan 03 '23 at 17:02

When using ops / graphs / jobs in Dagster it's very important to understand that the code defined within a @graph or @job definition is only executed when your code is loaded by Dagster, NOT when the graph is actually executing. The code defined within a @graph or @job definition is essentially a compilation step that only serves to define the dependencies between ops - there shouldn't be any general-purpose python code within those definitions. Whatever operations you want to perform on data flowing through your job should take place within the @op definitions. So if you wanted to print the values of your list that is be input via a config schema, you might do something like

@op(config_schema={"table_name":list})
def read_tableNames(context):
    lst=context.op_config['table_name']
    context.log.info(f'-------------->',type(tableNames_frozenList'))
    context.log.info(f'-------------->{tableNames_frozenList}')

here's an example using two ops to do this data flow:

@op(config_schema={"table_name":list})
def read_tableNames(context):
    lst=context.op_config['table_name']
    return lst
@op
def print_tableNames(context, table_names):
    context.log.info(f'-------------->',type(table_names)

@job
def simple_flow():
    print_tableNames(read_tableNames())

Have a look at some of the Dagster tutorials for more examples

how to iterate over a list of values returning from ops to jobs in dagster

1 Answers1