0

Parameters

Source Settings

Optimize

We have a generic dataflow that works for many tables, the schema is detected at runtime. We are trying to add a Partition Column for the Ingestion or Sink portion of the delta.

We are getting error: Azure Data Factory DataFlow Error: Key partitioning does not allow computed columns Job failed due to reason: at Source 'Ingestion'(Line 7/Col 0): Key partitioning does not allow computed columns

Can we pass the partition column as a parameter to a generic dataflow?

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
GVFLUSA
  • 25
  • 4

1 Answers1

1

Can we pass the partition column as a parameter to a generic dataflow?

I tried your scenario and got similar error.

enter image description here

There is a limitation of key partition method is we cannot apply any calculation to the partition column while declaring it. Instead, this must be created in advanced, either using derived column or read in from source.

To resolve this, you can try following steps -

  • First, I created a pipeline parameter with datatype string and gave column name as value. enter image description here

  • Click on Dataflow >> Go to Parameter >> In value of parameter select Pipeline expression >> and pass the above created parameter. enter image description here

OUTPUT:

It is taking it as partition key column and partitioning data accordingly. enter image description here

Reference : How To Use Data Flow Partitions To Optimize Spark Performance In Data Factor

Pratik Lad
  • 4,343
  • 2
  • 3
  • 11