0

I am building a prototype pipeline that does two things:

  1. (solid) Clears files out of an existing directory
  2. (solid) Runs a batch process to dump data into that directory.

Step #1 is all side-effect, and has no output to pass to #2. Is it possible to express a dependency between these two solids in a pipeline?

drewh
  • 10,077
  • 7
  • 34
  • 43

1 Answers1

2

I think the following code snippet from https://docs.dagster.io/examples/nothing should work for your use case:

from dagster import Nothing

@solid
def create_table_1(_) -> Nothing:
    get_database_connection().execute("create table_1 as select * from some_source_table")


@solid(input_defs=[InputDefinition("start", Nothing)])
def create_table_2(_):
    get_database_connection().execute("create table_2 as select * from table_1")


@pipeline
def my_pipeline():
    create_table_2(create_table_1())
Catherine Wu
  • 356
  • 1
  • 6