3

I would like to employ the Bourne shell background ampersand as part of ONE bsub execution command.

Note that I don't want to use a .bash file but rather one bsub command line for executing the entire procedure.

Note that I also don't want to split my bsub content into a few bsub commands that wait upon one another, but rather to wrap into a single bsub command line a few internal unix commands that wait upon one another using the Bourne shell background ampersand.

the following attempt only produced the output of the first internal command (the liftOver command), but neither sort nor mv were executed.

bsub -q medium -J $i "liftOver $i mm10ToMm9.over.chain.gz ${i/.bed/.mm9.bed} ${i/.bed/.unlifted_to_mm9.bed} & sort -k1,1 -k2,2n ${i/.bed/.mm9.bed} > ${i/.bed/.mm9.bed}.srt & mv ${i/.bed/.mm9.bed}.srt ${i/.bed/.mm9.bed}"

Outside of the bsub this procedure could be written as follows:

liftOver $i mm10ToMm9.over.chain.gz ${i/.bed/.mm9.bed} ${i/.bed/.unlifted_to_mm9.bed} & 
sort -k1,1 -k2,2n ${i/.bed/.mm9.bed} > ${i/.bed/.mm9.bed}.srt & 
mv ${i/.bed/.mm9.bed}.srt ${i/.bed/.mm9.bed}
user1934428
  • 19,864
  • 7
  • 42
  • 87
Roy
  • 723
  • 2
  • 8
  • 21
  • You could separate the individual commands by ';', and if I remember right, `bsub` also allows the command to be read from stdin, which means that you could pipe in the three lines. However, why do you want to execute the three commands in parallel???? It doesn't make sense to start the `mv`, before the `sort` is not finished. – user1934428 Feb 23 '16 at 08:05
  • ok, my wrong. the three command were actually separated by the ";" within the bsub wrap-up but although they have been separated by ";" they started to run in parallel, while I want them to wait to one another. I'm not sure that I can pipe always all my commands (as in some cases the some commands require generation of output file). I want to avoid starting mv before sort. – Roy Feb 23 '16 at 08:23
  • 1
    Question: What's `bsub`? What's difference with "cron's `batch`" command? – F. Hauri - Give Up GitHub Feb 23 '16 at 08:27
  • 1
    @Roy: The semicolon says that the commands start sequentially. You could also use a `&&` as a separator, which means that the following command is only executed if the previous one has exit code 0. If you use a `&`, you put the commands in the background and they run in parallel. – user1934428 Feb 23 '16 at 13:15
  • @F.Hauri: This is a command which belongs to a commercial product called LSF. You submit a job to a farm of processors, and LSF tries to do some load balancing. – user1934428 Feb 23 '16 at 13:16
  • @user1934428 Ok, I've found some docs... I think [my answer](http://stackoverflow.com/a/35572487/1765658) could be the *trick* – F. Hauri - Give Up GitHub Feb 23 '16 at 13:19
  • 1
    @F.Hauri: Great, I didn't know about `<<<`, and always had used `echo ....|` in such a case. Very convenient! – user1934428 Feb 24 '16 at 07:32

1 Answers1

1

I don't know bsub but I could maybe give you a trick I use with batch, ssh and some other differed shells:

bsub -q medium -J $i <<<"liftOver $i mm10ToMm9.over.chain.gz ${i/.be/.mm9.bed} ${i/.bed/.unlifted_to_mm9.bed} &&
     sort -k1,1 -k2,2n ${i/.bed/.mm9.bed} > ${i/.bed/.mm9.bed}.srt &&
     mv ${i/.bed/.mm9.bed}.srt ${i/.bed/.mm9.bed}"

Maybe you have to change double quotes (") for quotes (') if $i have to be developped by bsub (again, I don't know this product)

or

bsub -q medium -J $i <<-eocmd
    liftOver $i mm10ToMm9.over.chain.gz ${i/.be/.mm9.bed} \
         ${i/.bed/.unlifted_to_mm9.bed} &&
    sort -k1,1 -k2,2n ${i/.bed/.mm9.bed} > ${i/.bed/.mm9.bed}.srt &&
    mv ${i/.bed/.mm9.bed}.srt ${i/.bed/.mm9.bed}
        eocmd

(warn: only a tabulation $'\t' could be placed before word eocmd)

Again, you may have to escape $ sign if $i have to be developped by bsub.

For working around $TERM environment not set, I use sometime:

bsub -q medium -J $i /bin/bash <<-eocmd
    ...
        eocmd

In the hope this could do the job.

F. Hauri - Give Up GitHub
  • 64,122
  • 17
  • 116
  • 137
  • thanks alot @F. Hauri. Your solution works for me! This is the final version I end up using in my code now: *note that indeed the second command - awk - where ever dollar sign or ( " ) signs are used they must be escaped by ( \ ) Exactly the solution I looked for! Tnx bsub -q medium -J $i <<<"liftOver $i mm10ToMm9.over.chain.gz ${i/.bedGraph/.mm9.bedGraph} ${i/.bedGraph/.unlifted_to_mm9.bedGraph} && sort -k1,1 -k2,2n -k3,3n ${i/.bedGraph/.mm9.bedGraph}| awk '!x[\$1,\$2,\$3]++' FS=\"\\t\" > ${i/.bedGraph/.mm9.srt.bedGraph} && mv ${i/.bedGraph/.mm9.srt.bedGraph} ${i/.bedGraph/.mm9.bedGraph}" – Roy Feb 23 '16 at 20:07