Python Azure Batch - Permission Denied Linux node

Question

I am running a python script on several Linux nodes (after the creation of a pool) using Azure Batch. Each node uses 14.04.5-LTS version of Ubuntu.

In the script, I am uploading several files on each node and then I run several tasks on each one of these nodes. But, I get a "Permission Denied" error when I try to execute the first task. Actually, the task is an unzip of few files (fyi, the uploading of these zip files went well). This script was running well until last weeks. I suspect an update of Ubuntu version but maybe it's something else.

Here is the error I get :

error: cannot open zipfile [ /mnt/batch/tasks/shared/01-AXAIS_HPC.zip ] Permission denied unzip: cannot find or open /mnt/batch/tasks/shared/01-AXAIS_HPC.zip,

Here is the main part of the code :

credentials = batchauth.SharedKeyCredentials(_BATCH_ACCOUNT_NAME,_BATCH_ACCOUNT_KEY)
batch_client = batch.BatchServiceClient(
        credentials,
        base_url=_BATCH_ACCOUNT_URL)
create_pool(batch_client,
                _POOL_ID,
                application_files,
                _NODE_OS_DISTRO,
                _NODE_OS_VERSION)
helpers.create_job(batch_client, _JOB_ID, _POOL_ID)
add_tasks(batch_client,
              _JOB_ID,
              input_files,
              output_container_name,
              output_container_sas_token)

with add_task :

def add_tasks(batch_service_client, job_id, input_files,
          output_container_name, output_container_sas_token):

print('Adding {} tasks to job [{}]...'.format(len(input_files), job_id))

tasks = list()

for idx, input_file in enumerate(input_files):
      command = ['unzip -q $AZ_BATCH_NODE_SHARED_DIR/01-AXAIS_HPC.zip -d $AZ_BATCH_NODE_SHARED_DIR',
                 'chmod a+x $AZ_BATCH_NODE_SHARED_DIR/01-AXAIS_HPC/00-EXE/linux/*',
                 'PATH=$PATH:$AZ_BATCH_NODE_SHARED_DIR/01-AXAIS_HPC/00-EXE/linux',    
                 'unzip -q $AZ_BATCH_TASK_WORKING_DIR/'
                 '{} -d $AZ_BATCH_TASK_WORKING_DIR/{}'.format(input_file.file_path,idx+1),                   
                 'Rscript $AZ_BATCH_NODE_SHARED_DIR/01-AXAIS_HPC/03-MAIN.R $AZ_BATCH_TASK_WORKING_DIR $AZ_BATCH_NODE_SHARED_DIR/01-AXAIS_HPC $AZ_BATCH_TASK_WORKING_DIR/'
                  '{} {}' .format(idx+1,idx+1),                     
                 'python $AZ_BATCH_NODE_SHARED_DIR/01-IMPORT_FILES.py '
                  '--storageaccount {} --storagecontainer {} --sastoken "{}"'.format(
                   _STORAGE_ACCOUNT_NAME,
                   output_container_name,
                   output_container_sas_token)]          
      tasks.append(batchmodels.TaskAddParameter(
            'Task{}'.format(idx),
            helpers.wrap_commands_in_shell('linux', command),
            resource_files=[input_file]
            )
      )
Split = lambda tasks, n=100: [tasks[i:i+n] for i in range(0, len(tasks), n)]
SPtasks = Split(tasks)      
for i in range(len(SPtasks)):
    batch_service_client.task.add_collection(job_id, SPtasks[i])

Do you have any insights to help me on this issue? Thank you very much.

Robin

Questions seeking debugging help ("why isn't this code working?") must include the desired behavior, a specific problem or error and the shortest code necessary to reproduce it in the question itself. Questions without a clear problem statement are not useful to other readers. See: How to create a Minimal, Complete, and Verifiable example. — yacc, Nov 24 '17 at 12:45

score 1 · Answer 1 · answered Nov 27 '17 at 00:56

looking at the error, i.e.

error: cannot open zipfile [ /mnt/batch/tasks/shared/01-AXAIS_HPC.zip ]
Permission denied unzip: cannot find or open /mnt/batch/tasks/shared/01-AXAIS_HPC.zip,

This seems like that the file is not present at the current shared directory location or it is is not in correct permission. The former is more likely.

Is there any particular reason you are using the shared directory way? also, How are you uploading the file? (i.e. hope that the use of async and await is correctly done, i.e. there is not greedy process which is running your task before the shared_dir stuff is available to the node.)
side note: you own the node so you can RDP / SSH into the node and find it out that the shared_dir are actually present.

Few things to ask will be: how are you uploading these zip files.

Also if I may ask, what is the Design \ user scenario here and how exactly you are intending to use this.

Recommendation:

There are few other ways you can use zip files in the azure node, like via resourcefile or via application package. (The applicaiton package way might suite it better to deal with *.zip file) I have added few documetns and places you can have a look at the sample implementation and guidance for this.

I think a good place to start are: hope material and sample below will help you. :)

Also I would recommend to recreate your pool if it is old which will ensure you have the node running at the latest version.

Python Azure Batch - Permission Denied Linux node

1 Answers1