Run simultaneous tasks followed by single task in one slurm script

Question

I need a Slurm script to do the following:

1. Create a directory in scratch space for temporary storage (requested using sbatch --gres disk:1024)
2. Run samtools sort on hundreds of bam files and store sorted copies in scratch space (as many at once as possible)
3. After sorting, run samtools index on sorted files in scratch space (as many at once as possible)
4. After indexing, run a single (large) task using all sorted/indexed bam files (more CPUs the better)
5. Copy needed files back to main storage system and delete remaining files (including sorted bam and index files)

Using basic job arrays doesn't seem to work as it throws off the steps that only need to be performed once. The single task at the end says the files aren't there so I'm guessing the script was getting ahead of itself and deleting everything before others finished (maybe running rm as many times as the arrays are used), so trying something else.

The following script gives me errors from samtools saying there is no such file or directory when trying to create the sorted bam files.

[E::hts_open_format] Failed to open file /mnt/scratch/parallel_build/Sample_13-00145.bam
samtools sort: failed to create "/mnt/scratch/parallel_build/Sample_13-00145.bam": No such file or directory

If I drop --nodes to 1, samtools sort works fine, however it only runs sequentially and after about 50 files or so it jumps ahead, runs part 2 on what files are there and the single task at the end can't find the rest of the files (works ok when less than 30 files or so are used).

Any help on how I can make this perform correctly would be great. I'd like to fit as many tasks as possible across all nodes when space becomes available for parts 1 and 2. Part 3 needs to be on one node, but greatly benefits from many CPUs so it would be nice to give that more CPUs than the small parallel tasks preceding it (those can have less CPUs if it means more tasks going at once). Keep in mind I do need to do this all in one process as the scratch space is required for various reasons.

#!/bin/sh
#SBATCH --job-name=majiq_build
#SBATCH --nodes=5
#SBATCH --tasks-per-node=4 # 4 tasks, would like more
#SBATCH --cpus-per-task=8
#SBATCH --time=30:00:00
#SBATCH --mem-per-cpu=4G
#SBATCH -A zheng_lab
#SBATCH -p exacloud
#SBATCH --error=/home/exacloud/lustre1/zheng_lab/users/eggerj/Dissertation/splice_net_prototype/beatAML_data/splicing_quantification/test_build_parallel/log_files/build_parallel.x80.%J.err
#SBATCH --output=/home/exacloud/lustre1/zheng_lab/users/eggerj/Dissertation/splice_net_prototype/beatAML_data/splicing_quantification/test_build_parallel/log_files/build_parallel.x80.%J.out

# Set variables
DIR=/home/exacloud/lustre1/zheng_lab/users/eggerj
TMP=/mnt/scratch/parallel_build
WORK=$DIR/Dissertation/splice_net_prototype/beatAML_data/splicing_quantification

# Create temporary directory in requested scratch space to store files
mkdir $TMP
mkdir $TMP/majiq_out

##################################################################################################################
#
# PART 1: Sort bam files (in parallel) and store in scratch space
#         (wait until all are finished before part 2)
#
##################################################################################################################
while read F  ;
do
    fn="$(rev <<< "$F" | cut -d'/' -f 1 | rev)"
    echo $fn
    srun -N 1 -n 1 -c $SLURM_CPUS_PER_TASK --exclusive /opt/installed/samtools-1.6/bin/samtools sort -@ $SLURM_CPUS_PER_TASK -m 4G -o $TMP/$fn $F &
done <$WORK/test_bams/test_bam_list_x80.txt
wait

##################################################################################################################
#
# PART 2: Index bam files (in parallel) in scratch space
#         (wait until all are finished before part 3)
#
##################################################################################################################
for file in $TMP/*bam ;
do
    srun -N 1 -n 1 -c $SLURM_CPUS_PER_TASK --exclusive /opt/installed/samtools-1.6/bin/samtools index -@ $SLURM_CPUS_PER_TASK $file &
done
wait

# Check files actually made it before running MAJIQ
ls -lh $TMP

##################################################################################################################
#
# PART 3: Run MAJIQ build (single task) using all bam files (after all have been indexed)
#
##################################################################################################################

# Activate majiq virtual environment
source $DIR/majiq/bin/activate

# Run MAJIQ build using all bam files (.ini file indicates that bam files are in temp directory)
srun -N 1 -n 1 -c $SLURM_CPUS_PER_TASK --exclusive majiq build $WORK/gtfs/Homo_sapiens.GRCh37.75.gff3 -c $WORK/test_build_parallel/settings.x80.parallel.ini \
                                                          -j $SLURM_CPUS_PER_TASK --output $TMP/majiq_out --min-experiments 0.25
wait

# Move majiq output files from tmp directory to output directory on Lustre and remove
cp $TMP/majiq_out/* $WORK/test_build_parallel/majiq_out_x80/
rm -r $TMP