Azure Batch Apps install python packages on startup

Question

We are using Azure Batch Apps which will create multiple VMs which can be used to run our tasks parallelly. We are using python for data fetching tasks.

We have mentioned in the batch apps to install anaconda on the VMs when they start up. Anaconda is installed properly. We have listed out the packages(requirements.txt) we need to install to run the tasks.

pip install -r requirements.txt

Some packages get installed correclty, but some packages result in the following error,

 Error [Error 6] The handle is invalid while executing command python setup.py egg_info
Exception:
Traceback (most recent call last):
  File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\basecommand.py", line 209, in main
    status = self.run(options, args)
  File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\commands\install.py", line 310, in run
    wb.build(autobuilding=True)
  File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\wheel.py", line 748, in build
    self.requirement_set.prepare_files(self.finder)
  File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_set.py", line 360, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_set.py", line 591, in _prepare_file
    abstract_dist.prep_for_dist()
  File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_set.py", line 127, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\req\req_install.py", line 430, in run_egg_info
    command_desc='python setup.py egg_info')
  File "C:\user\tasks\shared\anaconda2\lib\site-packages\pip\utils\__init__.py", line 678, in call_subprocess
    cwd=cwd, env=env)
  File "C:\user\tasks\shared\anaconda2\lib\subprocess.py", line 702, in __init__
    errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
  File "C:\user\tasks\shared\anaconda2\lib\subprocess.py", line 823, in _get_handles
    p2cread = _subprocess.GetStdHandle(_subprocess.STD_INPUT_HANDLE)
WindowsError: [Error 6] The handle is invalid

When we open the VM and give the same command, all packages get installed correctly.

I just wonder where the issue is.

I hit the same issue while installing below requirements: - cryptography - azure-batch - azure-storage — hula100, Dec 12 '16 at 06:49

score 0 · Answer 1 · answered Apr 11 '16 at 08:13

0

It seems that the issue was caused by some limits for Azure Batch service, you can see these limits here.

According to the error information, it seems that the installation process needs to fork the subprocess, but the maximum number of tasks per computer node is 4, please see below.

answered Apr 11 '16 at 08:13

Peter Pan

23,476
4
25
43

This limit is only in the context of scheduled tasks through the Batch API (i.e., AddTask) not the number of processes that can be forked/created on the node. Meaning the task scheduled itself can spawn any number of sub-processes. – fpark Sep 06 '16 at 16:43

Azure Batch Apps install python packages on startup

1 Answers1