I had an issue writing parquet file using pyspark (Koalas) with standalone cluster.
The error I encountered was java.io.IOException: Could not rename file.
I figured out from here that
it was because the driver ran by user, and executor processes are ran by root, and those roots did not have permission to write file on user folder.
My temporary solution was to save it into C:\ folder, suggested here.
However, I'm wondering if there's a way to configure pyspark to run executors by users as well so that I may able to write on the user folder.
