I am currently having issues with my Apache Spark Pool in Azure Synapse Analytics using python: A rather trivial task of displaying a pandas dataframe of 200k rows is not possible, but merging on 1.7 million rows and grouping is possible. This leaves me confused as I have not seen this issue previously.
Calling the dataframe with df_mara results in the following error:
LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID: d3082022-0cd9-4f87-8818-9edf2718faf5.
For info the shape of df_mara is ~200k rows and 17 columns with a size of ~40 mb.
I have tried to delete the spark pool and create a new one. I've tried turning up the nodes and the intelligent cache parameters, but no success. And since I have worked with this same spark pool for some time with no issues, I don't expect increasing the number of cores will fix the issue.
