-2

I'm new to Palantir Foundery. I need to know how to automate the data pipeline? Is that same if we do the data ingestion to foundry while setting schedule at data sync? OR after all done (such as coding to data transformation etc.) when we see the data lineage at that point we can do the scheduling to the data pipeline. Is that both same ? Is that the meaning to say automate the data pipeline? Please some one can explain me. Thank you

user
  • 145
  • 1
  • 2
  • 12

1 Answers1

1

It is hard to understand what is best suited if the context of the ask is unclear to you in the first place, but usually "automating a data pipeline" can indeed refer to scheduling.

Docs about Schedulers can be found here. Schedulers are operating on Datasets. Datasets can be filled with data ingested from other systems or generated via transforms (defined in many ways: code repositories, pipeline builder, etc.).

ZettaP
  • 719
  • 7
  • 11
  • Thanks for the answer, but still I'm wondering how to automate the data pipeline. If I need to automate the data pipeline, 1. Create a schedule at the place of Data Sync 2. Create a schedule at the Data Lineage . what I understood that. Is that the two ways? – user Mar 15 '23 at 11:54
  • 1
    You set schedules on datasets, which means that you set a schedule on the dataset which is the output of your Sync (and so, by building the dataset, Foundry will execute the Sync) and (potentially even the same schedule) on other datasets in the data lineage (for instance, downstream datasets from the dataset where the sync drops data in) – ZettaP Mar 16 '23 at 08:41