Cannot get Python shutil.copytree ignore pattern to work

Question

I am somewhat new to Python but not coding. I'm trying to get copytree to work, and it does, but it will not work with ignore_patterns. I think it's because I don't know how to send a variable list of files to the ignore_patterns() function. I tried a list and a string. It says the list won't work because it's not immutable. The string, I finally realized, does not match anything because it is a CSV list essentially. So how do I get the parameters to the ignore_patterns?

I am using Python 2.7 on Linux (AWSLinux). I've imported all the packages so here is the function itself I wrote:

    def copy_user_data(shadow_data):
        user_data = shadow_data.split(EOL)
        files = ['file1','.file12']
        dirs = ['dir1','.dir12']

        for datum in user_data:
            user = datum.split(":")[0]
            user_home = datum.split(":")[5]

            if os.path.exists(user_home):
                ignore_list = []
                for file in os.listdir(user_home):
                    if file not in files and file not in dirs:
                        ignore_list.append(file)
                ignore_string = "'"+"','".join(ignore_list)+"'"
                logger.info("Copying "+user_home)
                logger.info("ignoring:"+ignore_string)
                dest = os.path.join(TRANSPORT_DIRECTORY,user)
                shutil.copytree(user_home,dest,ignore=shutil.ignore_patterns(ignore_string))

Per the first comment, this is a smaller example. it will require users to exist.

NOTE: There are no error messages when I use a string. The copytree copies ALL files and directories. If I list, specifically, the parameters in ignore_patterns (e.g. ignore_patterns("useless.txt","report.pdf")) it works and ignores those files. But it does not ignore them with the parameters I am passing in. Essentially, I don't know what data structure to pass in to get this to work. I've tried a string in the format "file1, file2, file3". I've tried a list ["file1","file2"], but I get:

    2017-03-27 11:12:01,748 - __main__ - CRITICAL - UNKNOWN ERROR:<type 'exceptions.Exception'>
    Traceback (most recent call last):
      File "./user_migration.py", line 133, in main
        copy_user_data(passwd_data)
      File "./user_migration.py", line 89, in copy_user_data
        shutil.copytree(user_home,dest,ignore=shutil.ignore_patterns(ignore_list))
      File "/usr/lib64/python2.7/shutil.py", line 173, in copytree
        ignored_names = ignore(src, names)
      File "/usr/lib64/python2.7/shutil.py", line 141, in _ignore_patterns
        ignored_names.extend(fnmatch.filter(names, pattern))
      File "/usr/lib64/python2.7/fnmatch.py", line 51, in filter
        re_pat = _cache[pat]
    TypeError: unhashable type: 'list'

Smaller version

    import shutil
    import os


    users = ["bob","joe","list"]
    files = ['.bashrc','.bash_profile']
    dirs = ['.ssh','.google_authenticator']

    for user in users:
        user_home = os.path.join("/home",user)
        if os.path.exists(user_home):
            ignore_list = []
            for file in os.listdir(user_home):
                if file not in files and file not in dirs:
                    ignore_list.append(file)
            ignore_string = "'"+"','".join(ignore_list)+"'"
            dest = os.path.join("/var/tmp/move",user)
            shutil.copytree(user_home,dest,ignore=shutil.ignore_patterns(ignore_string))

Please create a [Minimal, Complete, and Verifiable example](https://stackoverflow.com/help/mcve) — Taku, Mar 27 '17 at 14:34
FYI I've also tried to set() the list but still get the unhashable type error. — Jeff Masiello, Mar 27 '17 at 15:18
also tried sending it a tuple to no avail. no error. just does not ignore stuff. — Jeff Masiello, Mar 27 '17 at 15:42
One piece of knowledge is that tuple is an immutable sequence. As for the remaining dilemma, [shutil: copytree example](https://docs.python.org/2/library/shutil.html#copytree-example), explains how to use `ignore_files` (note that it receives a tuple of strings). — CristiFati, Mar 27 '17 at 16:01
@JeffMasiello: don't put an answer in your question. You can post an answer to your question and then accept it. — DSM, Mar 27 '17 at 16:12

score 0 · Answer 1 · answered Mar 27 '17 at 19:04

(Posted on behalf of the OP).

I did not realize the ignore parameter needed to be a callable function accepting 2 parameters. I altered that and returned a tuple and it worked. I'd been stuck for 2 days.

    def copy_user_data(user_data):

        for datum in user_data:
            user = datum.split(":")[0]
            user_home = datum.split(":")[5]
            if os.path.exists(user_home):
                dest = os.path.join(TRANSPORT_DIRECTORY,user)

                shutil.copytree(user_home,dest,ignore=ignore_files)

    def ignore_files(path, names):
        files = ['.bashrc','.bash_profile','.ssh','.google_authenticator']
        ignore_list = []
        for file in os.listdir(path):
            if file not in files:
                ignore_list.append(file)
        return tuple(ignore_list)

Cannot get Python shutil.copytree ignore pattern to work

1 Answers1