1

I want to be able to unzip an input file before running tasks to reduce data transfer in my batching process. To do this I added the following command line to my JobPreparationTask:

CommandLine = "/bin/bash -c 'DEBIAN_FRONTEND=noninteractive apt-get install -y unzip; unzip $AZ_BATCH_TASK_WORKING_DIR//myInput.zip'"

But instead of looking at my task working directory where I have my file (job/task/wd/myInput.zip) is looking at the job preparation folder and I get the following error:

"cannot find or open /mnt/batch/tasks/workitems/myJob/job-1/jobpreparation/wd//myInput.zip"

I have tried using other environmental paths but I always get similar errors. How can I access the job/task/wd to unzip the file?

SandiaDeDia
  • 291
  • 2
  • 9

1 Answers1

2

I think its the env var which you are using is wrong, below are some recommendation as well answer to your current way here:

  • I assume that your zip file is part of resourceFiles in this case? But at what context?
  • Refer to resource files here: https://learn.microsoft.com/en-us/azure/batch/resource-files

  • In-particular at the part where it say: All types of tasks support resource files: tasks, start tasks, job preparation tasks, job release tasks, etc so at what point your resource files get uploaded under what context?

  • If upload is happening at the JobPrep and current task is referring to the JobPrep then that might explain this behavior.

  • if the Zip is part of the resoource files at task level then cmdline should be part of those tasks and the env var should be using on the current context.

  • If not: then make sure zip is part of resroucefiles with jobPrep and use the environment vars accordingly.

  • Regarding env var why are you not using : AZ_BATCH_JOB_PREP_WORKING_DIR or AZ_BATCH_JOB_PREP_DIR ? Please refer here : https://learn.microsoft.com/en-us/azure/batch/batch-compute-node-environment-variables#environment-variables which will allow access to your job prep dir level.

Extra

Idea share just so that you are aware and to keep in mind and only use if this ever fits your need. (Only use it if you have read the article and it actually fits your design scenario otherwise please ignore and above should resolve your issue) :)

An application package is a .zip file that contains the application binaries and supporting files that are required for your tasks to run the application. Each application package represents a specific version of the application.

You can specify application packages at the pool and task levels.

Hope this fits your need. :) thanks

Tats_innit
  • 33,991
  • 10
  • 71
  • 77
  • My file is in **job/task/wd/myInput.zip** so I thought AZ_BATCH_TASK_WORKING_DIR was the best options. I have tried all of them anyway and when I use $AZ_BATCH_JOB_PREP_WORKING_DIR I get the exact same path and error as AZ_BATCH_TASK_WORKING_DIR (dont know why they represent the same path) . Thanks. – SandiaDeDia Apr 15 '19 at 02:56
  • Ah, Cool so to best understand, is your resource files is part of task level resource file? i.e. how are you uploading the resoruce files and under what context? task level or something else **Then** why your job prep command line is using the `jobprep` location context? i.e. why not those normal task with context of `AZ_BATCH_TASK_WORKING_DIR` ? – Tats_innit Apr 15 '19 at 02:58
  • continue... Essentially, your reply to "`when the resource files gets added`" will clear most of the thing in this scenario, i.e. if you run this `commandLine` in the context of `task level` then the env var will get you that location of every task. otherwise at the `JobPrep` task the working location of the task is same as the `JobPrep` task right? I have a feeling that there is some minor diconnect hear, please feel free to add more and I should be able to help you :) – Tats_innit Apr 15 '19 at 03:08
  • More... so for clarity, read this, https://learn.microsoft.com/en-us/azure/batch/batch-api-basics#files-and-directories the context of `AZ_BATCH_TASK_WORKING_DIR` is at the context of "**Task working Directory**" in case of `JobPrep` it should be the `jobPrep` task directory which it points to, as you mention that the command line is run as part of `JobPrep` **further** `Within each task directory, the Batch service creates a working directory (wd) whose unique path is specified by the AZ_BATCH_TASK_WORKING_DIR environment variable.` – Tats_innit Apr 15 '19 at 03:19
  • 1
    Resource file was at task level. I changed the command line from jobpreparation to Task and **now I am able to unzip the file*** thanks!!!. Now I have all my input files in the **task/wd/** as I wanted but the exe application is not able to see the unziped ones. Should I add the unzipped files as a resource file to the task? – SandiaDeDia Apr 15 '19 at 03:24
  • Awesome `:)` Glad it helped you. Yep, you know your scenario better so if its task level for all then `unzip` them when the file is available to your. I would also recommend to read about `application package` feature which will get your files and unzip them at a specific location for all tasks. `:)` and welcome to SO. – Tats_innit Apr 15 '19 at 03:27