Has anyone successfully used shopt -s extglob (extended globbing) in bash with SBATCH settings on a HPC?

Question

To summarise:

I am using bash shell, version: 4.2.46(2)-release

I want to submit a batch job script to slurm job scheduler where, in the script I use extended globbing which is turned on using shopt -s extglob on a separate line to the extended globbing command I am trying to use.

I want to run these commands:

echo !(wmparc.nii.gz)
#  for sid in $SubIdStr; do

#     cd $sid
#     rm -r rfMRI T2w
#     cd T1w
#     rm -r !(wmparc.nii.gz)
#     cd ..
#     cd MNINonLinear/
#     rm -r !(wmparc.nii.gz|T1w_restore_brain.nii.gz|Results)
#     cd ..
#     cd ..

#  done

I have tried having shopt -s extglob before these commands with a newline between them, eg. :

shopt -s extglob

echo !(wmparc.nii.gz)

This does not work and when writing shopt extglob after, there is no display of extended globbing being on or off that is output to my log files.

The only place I have found shopt -s extglob to work is right at the start, eg:

#!/bin/bash
shopt -s extglob

This is a problem, however, because my SBATCH settings are not interpreted, eg. :

#SBATCH --mail-type=END                         
#SBATCH --mail-user=
#SBATCH --partition=imgcomputeq
#SBATCH --qos=img
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --mem=5g
#SBATCH --time=50:00:00

This is a description of sbatch from the manual:

sbatch submits a batch script to Slurm. The batch script may be given to sbatch through a file name on the command line, or if no file name is specified, sbatch will read in a script from standard input. The batch script may contain options preceded with "#SBATCH" before any executable commands in the script. sbatch will stop processing further #SBATCH directives once the first non-comment non-whitespace line has been reached in the script.

The directories in which I am trying to delete everything but the files contained within the extended glob there are a mix of directories and files which I want to remove.

I'm curious if anybody has encountered this issue before/knows of a workaround? Or can provide a alternate solution?

score 0 · Answer 1 · answered Apr 30 '20 at 08:38

Workaround that is more POSIX (compatible) and therefore, arguable, better:

for sid in $storeIdStr; do

  rm -r $sid/rfMRI $sid/T2w
  find $sid/MNINonLinear/ -maxdepth 1 -mindepth 1 ! -name 'wmparc.nii.gz' ! -name 'T1w_restore_brain.nii.gz' ! -name 'Results' -exec rm -rf {} \;
  find $sid/T2w/ -maxdepth 1 -mindepth 1 ! -name 'T2w_acpc.nii.gz' -exec rm -rf {} \;
  find $sid/T1w/ -maxdepth 1 -mindepth 1 ! -name 'wmparc.nii.gz' -exec rm -rf {} \;

done

Would still be interested in hearing anybody's answers regarding why shopt -s extglob must be put at the start of the script or anybody else's experiences when using in conjunction with SBATCH settings?

Has anyone successfully used shopt -s extglob (extended globbing) in bash with SBATCH settings on a HPC?

1 Answers1