1

Usually to submit to condor I have to ssh to a machine which is inside the htcondor cluster and then run condor_submit. Now I would like to do the same, but from a machine outside the cluster (e.g. my laptop), where I don't have condor (and condor_submit) installed or configured. In particular I would like to run in the Docker universe.

Everything that is needed to run is inside a Docker image. Input will be read from external resources (s3 buckets, xrootd, ...) and similarly for the output. Different jobs need to take a different file as input, or group of files. This step (map the inputs to the jobs) in principle can be done by the user.

Is it technically possible? For example it is possible to send the informations via ssh or http to the cluster? It would be better to avoid to ask the user to install software specific to the batch system.

Ruggero Turra
  • 16,929
  • 16
  • 85
  • 141

1 Answers1

1

This is technically possible, and there are more than one ways to do it.

But first, we need to know what you would like to do about data, both input and output. Are there per-job input files? If so, where do they come from? Are there per-job output files? If so, where would you like them to end up?

Also, can you install HTCondor on your laptop? It doesn't need to be running any of the daemons, just have condor_submit and the rest of the tools installed. If condor_submit is installed on your laptop, we can do what we call a "remote submit", and run this command locally on the laptop, but tell it to queue the job to some condor_schedd running on a remote machine, by passing the -name name_of_remote_schedd option to condor_submit. You'll need to set up security to do so, which might require the help of your administrator. Or, if your laptop is not running Windows, you can log into the schedd machine, get a token to allow you to submit, and use that on your laptop.

Otherwise, if you have password-less ssh setup, it is easy to just scp a submit file over to the submit machine, and run ssh condor_submit

(Edits written after questions above clarified in original post)

Also, this assumes that your laptop is not running Windows...

HTCondor version 9.0 and later supports it's own kind of tokens (based on jwt) that can grant access to a schedd. If you can log into the machine with the schedd, run the command

$ condor_token_fetch -token name_of_schedd

This assumes that your system administrator has set up a signing key on this machine, if they haven't, ask nicely that they do.

condor_token_fetch will create a token that duplicates the access you already have on this local machine, and writes it to ~/.condor/tokens.d/name_of_schedd

Copy this directory tree (securely!) to your home directory on your laptop, and install the appropriate HTCondor for your distribution.

Then, you should be able to run commands to query and submit to you schedd like

$ condor_q -pool name_of_cm_machine -name name_of_schedd
$ condor_submit -pool name_of_cm_machine -name name_of_schedd submit_file
Greg
  • 703
  • 3
  • 6
  • 1
    thanks, I have updated my OP. What kind of token? How to get it? Kerberos? – Ruggero Turra Mar 26 '23 at 12:58
  • 1
    thanks for the updated answer. Still it is not clear if I can submit a job from my laptop without install condor, e.g. sending a http request to a remote service which talk with condor. – Ruggero Turra Mar 26 '23 at 23:06
  • There is no out-of-the-box HTTP interface for submitting HTCondor jobs. This is relatively easy to do with the HTCondor Python bindings and something like flask, but usually sites that want to do this do not want a generic submit, but something very tuned to their environment. – Greg Mar 26 '23 at 23:36