0

I'm trying to run a script which will submit a bunch of awk commands along the following lines:

qsub -l h_vmem=32G -l h_rt=04:00:00 "awk '{if (last != $10) close(last); print >> "/directory/working/"$10 ".txt"; last = $10}' /directory/working/working_file.txt"

The awk command itself works fine when run in an interactive session, the issue is that when combined with qsub it returns something along the lines of:

error opening awk '{if (last != 0) close(last); print >> '/directory/working: No such file or directory

Initially I thought this would be solved by putting the entire awk command in double quotation marks (as I've done above) but this also returned the same error. My understanding of this is that it isn't working because of the extra double quotation marks inside the awk command. But as far as I'm aware, I need those extra double quotation marks inside the command.

I tried changing it so that every quotation inside the first set of double quotes were just single quotes, but that didn't work. I then tried:

qsub -l h_vmem=32G -l h_rt=04:00:00 "$(awk ...)"

But that did not seem to work either (or at least after submitting it, it didn't return an error but didn't actually seem to submit.

Is there any way around this/any solutions with all the various quotations?

Sabor117
  • 111
  • 1
  • 11
  • 1
    I dunno about qsub, try using `\"` instead of `"` for quotes required for print strings – Sundeep Mar 20 '19 at 15:58
  • 1
    In addition to Sundeep's comment: `$10` will expand (to the empty string) before being passed to `qsub` or even `awk`. Note that single quotes inside double quotes do not quote anything, they are just literal characters. Write `\$10` instead of `$10`. – Socowi Mar 20 '19 at 21:48

1 Answers1

1

Using multiple levels of quoting is likely to introduce bugs, so is best avoided.

It appears that qsub expects either the filename of a script or an executable program followed by arguments. However, the syntax is different in the two cases. See: Submitting Job Directly by Specifying Executable on Command Line

qsub scriptfile

qsub -- program arg1 arg2 arg3 ...

So the simplest solution seems to be not to nest quotes but to prepend --:

qsub -l h_vmem=32G -l h_rt=04:00:00 -- awk '{if (last != $10) close(last); print >> "/directory/working/"$10 ".txt"; last = $10}' /directory/working/working_file.txt

For longer commandlines, it may be better to save the script as a file, somewhere that qsub can find it, so you don't need to worry about quoting at all.

If you must nest quotes in other situations, you can get bash (v4.4+) to show you what is needed:

raw=$(cat <<'EOD'
awk '{if (last != $10) close(last); print >> "/directory/working/"$10 ".txt"; last = $10}' /directory/working/working_file.txt
EOD
)
quoted="${raw@Q}"
jhnc
  • 11,310
  • 1
  • 9
  • 26
  • I believe that this should have fixed the problem (and is a bit more elegant than the suggestion in the comments of using ```\"```). However, I suspect due to some foibles with my server when attempting to run this it basically seemed to refuse to do anything. As there were no errors (that I could discover) though, I do suspect this should have worked were there not some hang-ups my end. – Sabor117 Mar 21 '19 at 14:23