2

When I am working with Foundry, there are some options to configure my job profile: num_executors, driver_memory, executor_memory, etc. I am wondering which is the equivalence of these profiles to the worker types in AWS. If I use AWS Glue Studio, I can select G 1x (4vCPU, 16 GB RAM) or G 2x (8vCPU, 32 GB RAM). Additionally, I can select the max. number of workers.

Can someone help me to understand this?

Pablo Cosio
  • 71
  • 5

1 Answers1

0

You can go to the code repositories settings and load spark profiles to be available to your transforms.

Screenshot of the settings page with spark profile open

From there you can import the spark profiles you want, which include similar options. Then you can import into your transform using profiles https://www.palantir.com/docs/foundry/code-repositories/spark-profiles/

from transforms.api import configure

@configure(profile=['EXECUTOR_MEMORY_LARGE', 'EXECUTOR_CORES_MEDIUM', 'NUM_EXECUTORS_64'])
@transform(
    # your input dataset(s)
    my_input=Input("/path/to/input/dataset"),
    # your output dataset
    my_ouput=Output("/path/to/output/dataset"),
)
# your data transformation code
def my_compute_function(my_input):
    return my_input

Scroll through the spark profiles list and you should find other settings available to you. If any shows up locked, then you'll have to request your platform administrator to unlock it for you. It's often because they are expensive. i.e.: NUM_EXECUTORS_512 or EXECUTOR_CORES_EXTRA_LARGE

fmsf
  • 36,317
  • 49
  • 147
  • 195
  • 1
    Could you add which profile combinations would be comparable to "G 1x (4vCPU, 16 GB RAM) or G 2x (8vCPU, 32 GB RAM)"? – nicornk Jul 15 '22 at 12:52
  • So that's not really how it works. Instead of configuring the total box size, and then splitting by the workers, you just set the worker size directly. – fmsf Jul 15 '22 at 13:12
  • I am still confused regarding the equivalency. You say that it is not how it works, but could you give some additional details how it works? In AWS for example I can decide to have 8 workers of type G1x. At the end I have 32 vCPUs. Is some kind of comparison to Foundry profiles? 8 executors or 32 executors. G1x has 16 GB RAM, would this be Executor memory medium or large in Foundry? – Pablo Cosio Jul 16 '22 at 13:16
  • sorry, I don't have a direct comparison. – fmsf Jul 17 '22 at 16:40