2

I am submitting a pyspark job into EMR cluster from a AWS Step Function through apache livy. The pyspark job has Unix shell commands being fired.

Within test.py, 

subprocess.call(' echo $USER', shell=True, stdout=None, stderr=None)
subprocess.call(' mkdir /mnt/data', shell=True, stdout=None, stderr=None)

The log file has this output:

livy
mkdir: cannot create directory ‘/mnt/data’: Permission denied

The user "livy" is not able to create a directory even with sudo. I need this script to create a directory, how can it be done?

Parijat Bose
  • 380
  • 1
  • 6
  • 22

1 Answers1

0

User "livy" does not have permission to create directories in EMR.

But to create directories, we can use a bootstrap script. Or else, we can use "Steps" while creating EMR cluster.

While creating directories through bootstrap, sudo user privileges are required.

Parijat Bose
  • 380
  • 1
  • 6
  • 22