1

While creating DLT pipeline, we do not specify any cluster. Does DLT pipeline automatically spin up clusters. If yes, what type of clusters does it spin up?

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Rajib Deb
  • 1,496
  • 11
  • 30

1 Answers1

0

It depends on the cloud, but typically it selects general purpose computing. For example, on Azure it selected Standard_F8s, on AWS it's c5.2xlarge.

But you can specify your own node types by providing a cluster definition as a part of the pipeline (doc) - right now it should be done either by editing the JSON representation, or you can do that via Terraform as well.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • The dicumentation talks about a default and maintenance clusters. What does these two types signify – Rajib Deb Nov 21 '22 at 02:39
  • Default cluster is used for actual data processing. Maintenance cluster is used to perform operations like OPTIMIZE, VACUUM, ... – Alex Ott Nov 21 '22 at 11:41