Batch files created by python not running, but edit by notepad works

Question

I intend to use python and create a bunch of batch files, but the batch files created by python cannot be uploaded, while the same codes manually inputted can. I was wondering why.

My code is as follows

import os

os.chdir("c:/Users/alku/Documents/Work/ToDo/data")
## creating a folder to place submit codes
if not os.path.exists('submit'):
    os.makedirs('submit')

depvar = ["01", "02", "03", "04", "05", "06", "07", "08", 
          "09", "10", "11", "12", "13", "14", "15", "16", 
          "17", "18", "19", "20", "21", "22", "23", "24"]
##creating corresponding batch codes for supercomputer
###creating a master file
fmaster= open('submit/submit-R-00.csh',"wb+")
fmaster.write(bytes(
'#!/bin/csh' + "\n" +
'#SBATCH --time=00:10:00' + "\n" +
'#SBATCH --mem=6000' + "\n" +
'#SBATCH --cores=1' + "\n"
, encoding='utf8'))
fmaster.close()

for i in depvar:
    fmaster= open('submit/submit-R-00.csh',"ab+")
    f= open('submit/submit-R-' + str(i) + '.csh',"wb+")
    f.write(bytes(
    '#!/bin/csh' + "\n" +
    '#SBATCH --time=96:00:00' + "\n" +
    '#SBATCH --mem=256000' + "\n" +
    '#SBATCH --cores=8' + "\n" +
    'module load R/3.5.1' + "\n" +
    'Rscript alku-COVIDImpact031-quantregForest-part'+ str(i) + '.R'
    , encoding='utf8'))
    ##writing into the master file
    fmaster.write(bytes(
    'sbatch submit-R-' + str(i) + '.csh'  + "\n"
    , encoding='utf8'))

and the results are something like

#!/bin/csh
#SBATCH --time=96:00:00
#SBATCH --mem=256000
#SBATCH --cores=8
module load R/3.5.1
Rscript alku-COVIDImpact031-quantregForest-part01.R

With a master file submit-R-00.csh to submit them

#!/bin/csh
#SBATCH --time=00:10:00
#SBATCH --mem=6000
#SBATCH --cores=1
sbatch submit-R-01.csh
sbatch submit-R-02.csh
sbatch submit-R-03.csh
sbatch submit-R-04.csh
sbatch submit-R-05.csh
sbatch submit-R-06.csh
sbatch submit-R-07.csh
sbatch submit-R-08.csh
sbatch submit-R-09.csh
sbatch submit-R-10.csh
sbatch submit-R-11.csh
sbatch submit-R-12.csh
sbatch submit-R-13.csh
sbatch submit-R-14.csh
sbatch submit-R-15.csh
sbatch submit-R-16.csh
sbatch submit-R-17.csh
sbatch submit-R-18.csh
sbatch submit-R-19.csh
sbatch submit-R-20.csh
sbatch submit-R-21.csh
sbatch submit-R-22.csh
sbatch submit-R-23.csh
sbatch submit-R-24.csh

However, when I upload these files to our university supercomputer, the supercomputer almost cancel the jobs immediately. Furthermore, the system creates an all empty output, so I cannot figure out what happens.

I then typed the exact same codes with notepads, and upload them to the supercomputer. Somehow this time it works. These two versions are exactly the same, and I also use vim to check the line breaker. Although the problem is solved, I am very curious why the same codes created by python cannot be submitted, while typed manually can.

mukund ghode · Accepted Answer · 2021-01-03T06:21:13.397

1

in For loop, after the write is done, you have to close the files

fmaster.close()
f.close()

try closing the file and retry the best way to avoid missing f.close() is to use with, like:

with open("filename") as f:
    for line in f:
        # ... do stuff ...

this will automatically close the file. we dont need to explicity do f.close()

edited Jan 03 '21 at 06:21

answered Jan 03 '21 at 05:59

mukund ghode

252
1
7

Hi, thanks for your reply! It works! What happens when I didn't close() my file? The codes look the same. Sorry for my stupid question, I am new to coding... – alku Jan 03 '21 at 16:18

score 1 · Answer 2 · answered Jan 03 '21 at 06:21

1

use with to do operations on file. Eg.

with open("filename") as f:
    for line in f:
        # ... do stuff ...

to avoid explicitly closing the file

answered Jan 03 '21 at 06:21

xyz

21
3

Batch files created by python not running, but edit by notepad works

2 Answers2