3

I would like to run some python scripts that return some dictionaries on a cluster managed by HTCondor. I run the same on another cluster running MPI/Slurm and mpi4py and I was able to collect the output without saving files and transfer them to the main node. I cannot understand how to do it in HTCondor (even using python wrappers).

Is it possible to retrieve the output of python scripts in HTCondor? Do i need to use the schedd.retrieve("ClusterId == %d" % cluster) command? if so an example will be really appreciated.

Thanks!

s1mc0d3
  • 523
  • 2
  • 15

1 Answers1

0

You may specify a file to which you want to save your stdout in your submit file (see also Condor output file updating).

You can watch the output using htcondor_tail (not sure what the equivalent is in the python api).

Slightly off-topic: If you run python you can also use e.g. Dask to do your remote python stuff (see also Dask with HTCondor scheduler)

till
  • 570
  • 1
  • 6
  • 22
  • As you suggested I transitioned to dask-jobque to handle the jobs on top of HTCondor. It works really nice and make it easier to retrieve the output of the jobs. – s1mc0d3 Apr 28 '20 at 06:18