Recursively find and copy files from many folders

Question

I have some files in an array that I want to recursively search from many folders

An example of the filename array is ['A_010720_X.txt','B_120720_Y.txt']

Example of folder structure is as below which I can also provide as an array e.g ['A','B'] and ['2020-07-01','2020-07-12']. The "DL" remains the same for all.

C:\A\2020-07-01\DL C:\B\2020-07-12\DL

etc

I have tried to use shutil but it doesn't seem to work effectively for my requirement as I can only pass in a full file name and not a wildcard. The code I have used with shutil which works but without wildcards and with absolute full file name and path e.g the code below will only give me A_010720_X.txt

I believe the way to go would be using glob or pathlib which i have not used before or cannot find some good examples similar to my use case

    import shutil
    filenames_i_want = ['A_010720_X.txt','B_120720_Y.txt']
    RootDir1 = r'C:\A\2020-07-01\DL'
    TargetFolder = r'C:\ELK\LOGS\ATH\DEST'
    for root, dirs, files in os.walk((os.path.normpath(RootDir1)), topdown=False):
        for name in files:
            if name in filenames_i_want:
                print ("Found")
                SourceFolder = os.path.join(root,name)
                shutil.copy2(SourceFolder, TargetFolder)

osint_alex · Accepted Answer · 2021-07-25T01:22:39.823

1

I think this should do what you need assuming they are all .txt files.

import glob
import shutil

filenames_i_want = ['A_010720_X.txt','B_120720_Y.txt']
TargetFolder = r'C:\ELK\LOGS\ATH\DEST'
all_files = []
for directory in ['A', 'B']:
    files = glob.glob('C:\{}\*\DL\*.txt'.format(directory))
    all_files.append(files)
for file in all_files:
    if file in filenames_i_want:
        shutil.copy2(file, TargetFolder)

edited Jul 25 '21 at 01:22

answered Jul 25 '21 at 01:17

osint_alex

952
3
16

i have tried with the following but does not seem to work.Il show you my folder structure.Maybe i am missing something. I did not see use the ['2020-07-01','2020-07-12'] in your code as there is a sub folder with the date parameter. The file structure for the two files are Z:\A\2021-07-01\DL\A_010721_013738_0437_AT_B.txt.bz2 Z:\B\2021-07-01\DL\B_010721_183722_0643_AT_B.txt.bz2 I will put my code in the next comment – Kamikaze K Jul 25 '21 at 03:07
import glob import shutil filenames_i_want = ['A_010721_013738_0437_AT_B.txt.bz2','B_010721_183722_0643_AT_B.txt.bz2'] tra=['A','B'] TargetFolder = r'C:\ELK\LOGS\ATH\DEST' all_files = [] for directory in tra: files = glob.glob('Z:\{}\*\DL\*.bz2'.format(directory)) all_files.append(files) for file in all_files: if file in filenames_i_want: shutil.copy2(file, TargetFolder) – Kamikaze K Jul 25 '21 at 03:10
Tested with this but it doesn't give any error.Looks like it doesnt find the file. import glob import shutil filenames_i_want = ['70631','70632','test'] train=['7063'] TargetFolder = r'C:\ELK\LOGS\ATH\DEST' all_files = [] for directory in train: #files = glob.glob('C:\{}\2021-07-19\asts_data_logger\*.txt.bz2'.format(directory)) files=glob.glob('C:\ELK\LOGS\ATH\{}\2021-07-19\asts_data_logger\*.txt'.format(directory)) all_files.append(files) for file in all_files: if file in filenames_i_want: shutil.copy2(file, TargetFolder) print("found") – Kamikaze K Jul 25 '21 at 04:03

Recursively find and copy files from many folders

1 Answers1