2

I am new to databricks, I am confused after creating a cluster in databricks. Here databricks asked me to connect AWS account before creating a workspace and I did. Then I created a cluster. Now I want to know that, where does the cluster runs. Is it running on AWS ec2 instances or is it running on databricks workspace. What about driver and worker nodes?

Thanks in advance.

Utkarsh Pal
  • 4,079
  • 1
  • 5
  • 14
Hariharan B
  • 31
  • 4

1 Answers1

1

Databricks itself provides a management capabilities for clusters/jobs/users/... - these objects are defined in the Databricks environment (so-called "Control Plane"). Actual execution of the code happens on EC2 instances in your AWS account, either in VPC created by Databricks or in the VPC that you explicitly configured (doc) - it's referred as "Data Plane". You can find more information in the architecture overview documentation. Image from the documentation:

enter image description here

Alex Ott
  • 80,552
  • 8
  • 87
  • 132