Bash script to execute a python package on all files in a folder

Question

I am trying to run a python package called mETL from PuTTY and use it in all the files contained in a folder. The python package is mETL and I am using it to load data contained in 3 .csv files called upload-A.csv, upload-B.csv and upload-C.csv

Everything works perfectly when I do this process manually using the following commands:

metl -m migration.pickle -t new_migration.pickle -s folder_test/upload-A.csv config3.yml
metl -m migration.pickle -t new_migration.pickle -s folder_test/upload-B.csv config3.yml
metl -m migration.pickle -t new_migration.pickle -s folder_test/upload-C.csv config3.yml

All the data from every file is correctly uploaded or updated and the pickle files are updated accordingly.

But instead of doing this manually I want to make a loop that does this for all files contained in my 'folder_test/' folder, for which I tried the following Bash script:

folder_var=folder_test
for x in $folder_var
do
metl -m migration.pickle -t new_migration.pickle -s $x config3.yml
done

What happens after this is that the pickle files are created but no data is uploaded to the database.

can you format the commands that you are using in more readable way ,can't understand much from it. — g4ur4v, Sep 10 '14 at 13:18

g4ur4v · Accepted Answer · 2014-09-10T13:40:33.083

1

Try this

for x in folder_test/*
do
metl -m migration.pickle -t new_migration.pickle -s "${x}" config3.yml
done

edited Sep 10 '14 at 13:40

answered Sep 10 '14 at 13:21

g4ur4v

3,210
5
32
57

1) don't use `ls`, use `for x in folder_test/*`, 2) to protect filenames with spaces, make sure you quote `"$x"` wherever it's used – glenn jackman Sep 10 '14 at 13:38
Thanks for suggesting the changes. – g4ur4v Sep 10 '14 at 13:41

Bash script to execute a python package on all files in a folder

1 Answers1