1

I have the following files in a python pipeline

#in_para.py

nx = 31
c = 1

#solver.py
import numpy
import os   

def simple(output):
    ny = nx + 5
    c_2 = c + 2
    with open(os.path.join(output, 'Output.txt'), 'w') as f:
         print("Hello stackoverflow!", file=f)
         print("I have a question.", file=f)
        
if __name__=='__main__':
    simple()

#main.py 

import os
import numpy
import argparse

def main():
    parser = argparse.ArgumentParser() 
    parser.add_argument('-C','--Chk',type=str, help='Choose arg')
    parser.add_argument('-O','--output',type=str, default="./Output", help=' Provide the output Directory')

    args = vars(parser.parse_args())
 
    output = args['output']
    if not os.path.exists(output):
        os.makedirs(output)

    if args['Chk'] == 'compo1':
        simple(output)
    
if __name__=='__main__':
    main()

I would like to call the input file in_para.py through command line argument such that

python3 main.py -I in_para -C compo1 -O Ouput_dir

gives me the desired output:

this is a simple test 36
this is a simple test2 3

I know if I do from in_para import * it will solve the problem (in a different manner), but I would like to call the input parameters from the command line as a positional argument and pass it on to solver.py. I haven't been able to find an example in the documentation similar to above task. Besides, the above is just an example, in_para.py and solver.py has several input parameters and several lines of code, so I dont want the user of main.py to go into either file and modify it.

newstudent
  • 402
  • 6
  • 19
  • Try https://docs.python.org/3/library/argparse.html#fromfile-prefix-chars – Carlos Horn Jun 27 '22 at 08:19
  • Or try https://stackoverflow.com/questions/48538581/argparse-defaults-from-file – Carlos Horn Jun 27 '22 at 08:20
  • @CarlosHorn its not about having large set of input parameters, I would like to import it via an input file using argparse in `main.py` and pass it on to the `solver.py` – newstudent Jun 27 '22 at 08:24
  • You could also make your function in `solver.py` explicit and pass the input arguments like `def simple(nx, c)` this will allow you to implement any logic on providing the input, e.g. parse a yaml file which was given as command line argument. – Carlos Horn Jun 27 '22 at 08:29
  • Or is it important that the input file is a python file, e.g. because some parameters are computed there? – Carlos Horn Jun 27 '22 at 08:34
  • @CarlosHorn input file will just have the parameters, non of them are computed there. it can be any file extension. – newstudent Jun 27 '22 at 08:35
  • You are proposing to store configuration parameters as Python code. That is a bad idea. You're not writing Lua: there are better ways to do it. Look at `configparser` and `json` for more robust techniques. – BoarGules Jun 27 '22 at 08:43

1 Answers1

1

Change the implementation in solver.py

def simple(nx, c, output):  # provide parameters as function arguments
    ny = nx + 5
    c_2 = c + 2
    print("this is a simple test", ny)
    print("this is a simple test2", c_2)
    with open(os.path.join(output, 'Output.txt'), 'w') as f:
        print("Hello stackoverflow!", file=f)
        print("I have a question.", file=f)

In the main parse the parameters file

...  # other imports
import json

def main():
    parser = argparse.ArgumentParser() 
    parser.add_argument('-C','--Chk', type=str, help='Choose arg')
    parser.add_argument('-I','--input', type=argparse.FileType("r"), help='Input parameters')
    parser.add_argument('-O','--output', type=str, help='output dir')
    args = parser.parse_args()
    output = args.output
    # implement any way of parsing the input file, e.g. a json file
    params = json.load(args.input)
    
    ...  # do something else with the arguments
 
    if args.Chk == 'compo1':
        simple(output=output, **params)

...  # further code lines

Your inputs file would look like

{"nx": 31, "c": 1}

Comment on edit:

Either you use args = vars(parser.parse_args()) which results in a dictionary or you just use the args = parser.parse_args() to select arguments by attribute. In your original post you used the first approach which I adopted in the first answer. However, I would prefer not using vars. Note, that the code snippet to create the directory does not add value to the actual question of the post, so I removed it to avoid any distraction and added some placeholder instead.

Carlos Horn
  • 1,115
  • 4
  • 17
  • Note, this implementation uses a `json` file, but as @BoarGules suggested, you could also use an ini file parsed with `configparser` or even write your own parser (not recommended if json or ini files can do the job) another popular format is yaml, which is more focused on human readable than json, but it is not part of the standard python library. – Carlos Horn Jun 27 '22 at 08:57
  • May I know why you have modified the `solver.py` ? – newstudent Jun 27 '22 at 09:11
  • Because the function `simple` should receive the parameters as argument instead of getting them from the global namespace. – Carlos Horn Jun 27 '22 at 09:14
  • actually there shouldnt be nx in `solver.py` in the first place. that is a typo – newstudent Jun 27 '22 at 09:20
  • In any case, the point is to provide parameters as input arguments, so you have more flexibility on where you get them from. I would not recommend to work with globals here. – Carlos Horn Jun 27 '22 at 09:26
  • In other words, provide input to your functions as input arguments, this is what line number 2 in https://peps.python.org/pep-0020/#the-zen-of-python is all about. – Carlos Horn Jun 27 '22 at 09:34
  • Does the answer help you? Or do you really need a solution which dynamically sets the variables in the module namespace (like `from in_para import *`)? – Carlos Horn Jun 27 '22 at 13:03
  • I've got some idea, but I actually have `simple(output)` to save the output from `simple` function at a disired dir. so `simple(**params)` doesnt really work completely. My post is updated – newstudent Jun 27 '22 at 13:41
  • This is not a contradiction to the given answer, output would just be another argument of simple, note that `simple(**params)` is just a short hand for `simple(nx=params["nx"], c=params["c"], ...adding other keys in params)`, so you could easily add the output argument to the json file. – Carlos Horn Jun 27 '22 at 13:46