unable to convert from spark dataframe to AWS Glue dynamic frame

Question

I have a spark dataframe named cost_matrix. I am trying to convert this spark dataframe to a aws glue dynamic frame using the following line of code:

glue_cost_matrix = DynamicFrame.fromDF(cost_matrix, glueContext, 'glue_cost_matrix')

However, I'm getting this error:

An error occurred while calling z:com.amazonaws.services.glue.DynamicFrame.apply. java.lang.IllegalArgumentException

I am new at glue jobs so I'm not sure what it means. I would really appreciate your help. My Glue Job is a spark type and I'm using python as ETL language.

are you able to write cost_matrix to a CSV file? Can you confirm and also if possible provide the full script in your question — Prabhakar Reddy, Jun 03 '21 at 07:34

score 0 · Answer 1 · answered Jun 03 '21 at 05:13

It is likely that a closure is triggering when you convert the frame, and that the actual error is earlier, within some function your are calling on the frame. Are you using any udfs? It is likely the problem lies there.
The easiest way to find the cause would be to invoke something that will cause the closures to execute on your frame in various suspect locations, and then find the line number that is causing the error.
e.g. add something like:
cost_matrix.show()
After various lines, and see which line in blows up on in the glue logs. Presumably it will be before your FromDf.
The show() function will show the data in your frame, which will cause spark to execute any closures that it is delaying execution upon. Note that adding these lines will impact performance, so don't leave the various .show() functions in place after you find the source of the issue.

I found out that the error came before the conversion from spark df to dynamic df. — brenda, Jun 03 '21 at 18:40
Great, if you need help with the error, please update your question, otherwise is this the answer? — jonlegend, Jun 04 '21 at 06:03

unable to convert from spark dataframe to AWS Glue dynamic frame

1 Answers1