-1

I am totally new to this. I have to submit a job with TORQUE, and I want to run a python program that generates multiple files (millions). The program has already been written and it works fine. I also wrote a test script that prints 'Hello World' and submitted it, and it worked, generating a test.numbers that said 'Hello World'. I took the same script and instead of printing something, I wrote the name of the python program (which I made executable). However, the program finishes in a second (and it should take hours) and just generates a script.numbers with nothing in it. I suspect that my script doesn't actually run the python program. Any ideas or examples of a script with a python program that generates multiple files? Thanks

dc2
  • 157
  • 1
  • 1
  • 6
  • could you add the script to your question please? – Arnaud Potier Oct 16 '15 at 08:59
  • (1) make sure errors are captured e.g., add `1/0` to your script and find out where `ZeroDivisionError` is logged. (2) try a Python script that creates *one* file: `open('test.txt', 'w').write('hello')` and see how this file can be retrieved -- is there a persistent between job runs area where your script is allowed to save files? – jfs Oct 16 '15 at 09:05

1 Answers1

0

This is a very general question, so I can provide you only very general code and give you some hints.

Here is a small Python code snippet which you can take to create n files. You have to change it for your needs, of course.

for n in xrange(n):
    with open('file_' + str(n), 'w') as fw:
        fw.write('Hello World')

n is the number of files you want to create. The iteration step could look very different to you. Maybe you iterate over another list of items. So xrange(n) is just a broad example.

If you're creating more than one file in iteration steps, you have to name all the files differently, of course. So str(n) is added to the file name not to override already existing file. If you have different iteration (over specific items), you can name the files according to the items.

The with-statement for writing (opening) a file is a nice way not to have the needs to close the file. It will be done for your.

fw.write("text") has to be adjusted to your needs. Here, you can add complex functions which returns text which will be written to the file. You have to ensure the text can be written to file (str, unicode).

colidyre
  • 4,170
  • 12
  • 37
  • 53