How can I take a variable number of files as input for a python script?

Question

For example I want to be able to run this hypothetical command:

$ python script.py *.txt option1 option2

And have it execute on every file that matches *.txt

Currently I have only found information on operating on one file at a time

from sys import argv

self, file, option1, option2 = argv

perform_operation(file, option1, option2)

#function definition

Answer 1

You want to use the argparse-module:

import argparse

parser = argparse.ArgumentParser()

parser.add_argument("--option1")
parser.add_argument("--option2")
parser.add_argument("files", nargs="+")

opts = parser.parse_args()

print opts.option1
print opts.option2
print opts.files

Use like this:

 beer:~ deets$ python2.7 /tmp/argparse-test.py  text foo bar baz
 None
 None
 ['text', 'foo', 'bar', 'baz']

Answer 2

argv is a list. Let's assume that you are only going to pass filename arguments. If it's more complicated, then go with deets' answer.

self = sys.argv[0]
arguments = sys.argv[1:]

Now, arguments is a list of program arguments. Let's say we want to process them one at a time:

for argument in arguments:
    work(argument)

Or we want to pass all of them to a function:

work(arguments)

As to passing *.txt as an argument. Your shell (before your program even runs) will do most of the work for you.

If I run, python program.py *.txt where *.txt refers to 3 text files, then my shell will expand that such that my program will see python program.py a.txt b.txt c.txt.

Answer 3

multifile.py

"""
Usage:
    multifile.py <file>...
    multifile.py -h

Prints something about all the <file>... files.
"""

def main(files):
    for fname in files:
        print fname

if __name__ == "__main__":
    from docopt import docopt
    args = docopt(__doc__)
    files = args["<file>"]
    main(files)

Use it

Install docopt first

$ pip install docopt

Call the command without arguments:

$ python multifile.py
Usage:
    multifile.py <file>...
    multifile.py -h

Try help

$ python multifile.py -h
Usage:
    multifile.py <file>...
    multifile.py -h

Prints something about all the <file>... files.

Use it for one file:

$ python multifile.py alfa.py 
alfa.py

Use it for multiple files, using wildcards:

$ python multifile.py ../*.py

    ../camera2xml.py
    ../cgi.py
    ../classs.py

Conclusions

• docopt allows really many more options (see docopt)
• command line parsing can be easy in Python
  • argparse seems is standard part of Python since version 2.7
  • argparse can do a lot, but requires rather complex calls on many lines
  • plac is nice alternative, can server quickly in most cases
  • docopt seems to me to be the most flexible and at the same time shortest in required lines of code

Answer 4

Using inputfile from stdlib

There is one library in stdlib, which is often overlooked, called inputfile.

It by default handles all input on command line or from stdin as filenames and allows iterating not only over those files, but also over all the lines in it, modifying them, decompressing and many other practical things.

filenames.py - list all the filenames

import fileinput

for line in fileinput.input():
    print "File name is: ", fileinput.filename()
    fileinput.nextfile()

Call it:

$ python filenames.py *.txt
File name is: films.txt
File name is: highscores.txt
File name is: Logging.txt
File name is: outtext.txt
File name is: text.txt

upperlines.py - print all lines from multiple files in uppercase

import fileinput

for line in fileinput.input():
    print line.upper(),

and call it:

$ python upperlines.py *.txt
THE SHAWSHANK REDEMPTION (1994)
THE GODFATHER (1972)
THE GODFATHER: PART II (1974)
THE DARK KNIGHT (2008)
PULP FICTION (1994)
JAN HAS SCORE OF 101
PIETER HAS SCORE OF 900
CYRIL HAS SCORE OF 2
2014 APR 11  07:14:03.155  SECTORBLAH
   INTERESTINGCONTENT
   INTERESTING1 = 843
1. LUV_DEV <- HE'S A DEVELOPER
2. AMIT_DEV <- HE'S A DEVELOPER
....

upperlinesinplace.py - turn all lines in files into uppercase

import fileinput

for line in fileinput.input(inplace=True):
    print line.upper(),

Conclusions

• fileinput takes as default argument sys.argv[:1] and iterates over all files and lines
• you can pass your own list of filenames to process
• fileinput allows inplace changing, filtering, reading file names, line numbers...
• fileinput even allows processing compressed files