1

For very specific reasons in our use case we have to pass something as the value of system.profile.name property, when we execute a CloudDataFusionStartPipelineOperator from an Airflow DAG.

We use this property for selecting a DataProc cluster for DataFusion from our orchestration layer, but sometimes this value is deliberately not defined, this is when we want to fail back to default profile name on DataFusion side.

RawToRaw = CloudDataFusionStartPipelineOperator(
    # ...
    runtime_args = {
        # ...
        'system.profile.name': 'SYSTEM:<my_profile_name>',
    },

)

When we pass 'SYSTEM:<my_profile_name>' everything is fine. The profile (and the cluster) is selected properly.

Problem:
We have problems when the profile is not defined in our orchestration layer. We don't know what value to pass to DataFusion here. We can't avoid defining any value here for 'system.profile.name' property (due to some Airflow's + programming constraints).

Question:
What should we pass here as the value of 'system.profile.name' to surely fail back to the default profile set in DataFusion?

elaspog
  • 1,635
  • 3
  • 21
  • 51
  • Hi @elaspog, The runtime arguments are used to pick up certain compute profiles. Can you try running the pipeline without specifying the runtime arguments so that it will set the pipeline to use the default compute profile? – Vishal K Nov 17 '21 at 13:27
  • Hi @VishalK, As I've mentioned above, due to some constraints what are very specific to our DAG unfortunately we can't leave out this property. The reason of this that we define the structure of the `runtime_arguments` dictionary in the time when the DAG is generated by DAG a generator logic, but we want to fill the values of this dictionary in the DAG's execution time (after arguments are passed to the DAG via REST API). – elaspog Nov 17 '21 at 16:35
  • 2
    Hi @elaspog, unfortunately, as far as I can tell at the moment there is no value you can set which will revert the profile back to using the default. – Dennis Li Nov 20 '21 at 00:22

1 Answers1

0

As @Dennis Li has mentioned in the comments, at the moment there is no specific profile name that can be passed to revert the profile back to the default.

Kabilan Mohanraj
  • 1,856
  • 1
  • 7
  • 17