TypeError: cannot deepcopy this pattern object

Question

Trying to understand this error in my "Variable" class.

I was hoping to store a sre.SRE_Pattern in my "Variable" class. I just started copying the Variable class and noticed that it was causing all my Variable class instances to change. I now understand that I need to deepcopy this class, but now I run into "TypeError: cannot deepcopy this pattern object". Sure, I can store the pattern as a text string but the rest of my code already expects a compiled pattern! What would be the best way to copy my Variable class with a pattern object?

import re
from copy import deepcopy

class VariableWithRE(object):
    "general variable class"
    def __init__(self,name,regexTarget,type):
        self.name = name
        self.regexTarget = re.compile(regexTarget, re.U|re.M) 
        self.type = type 

class VariableWithoutRE(object):
    "general variable class"
    def __init__(self,name,regexTarget,type):
        self.name = name
        self.regexTarget = regexTarget
        self.type = type 

if __name__ == "__main__":

    myVariable = VariableWithoutRE("myName","myRegexSearch","myType")
    myVariableCopy = deepcopy(myVariable)

    myVariable = VariableWithRE("myName","myRegexSearch","myType")
    myVariableCopy = deepcopy(myVariable)

Given that compiler regexp is immutable, it's not necessary to deepcopy it. However I don't remember how tell `deepcopy()` how to handle a particular type (but note, that you *can* add attributes to the build in type if needed). — Jan Hudec, Jun 08 '11 at 13:22
What line from your code throwing exception? I have copied it and compile then -> it works without any errors at my side. The only thing i found is that you try to use variable that rewrite python method 'type' and this is not very good style. — Artsiom Rudzenka, Jun 08 '11 at 13:24

score 11 · Answer 1 · answered Jun 08 '11 at 13:26

11

deepcopy doesn't know anything about your classes and doesn't know how to copy them.

You can tell deepcopy how to copy your objects by implementing a __deepcopy__() method:

class VariableWithoutRE(object):
   # ...
   def __deepcopy__(self):
      return VariableWithoutRE(self.name, self.regexTarget, self.type)

answered Jun 08 '11 at 13:26

sth

222,467
53
283
367

The problem is to copy `VariableWithRE`, so your example doesn't really help, but it goes in the right direction. – Jochen Ritzel Jun 08 '11 at 15:43
I tried this method in both of my classes, first I got "TypeError: __deepcopy__() takes exactly 1 argument (2 given)", then I looked at [this](http://stackoverflow.com/questions/3036648/how-can-i-override-deepcopy-in-python) question, and tried "def __deepcopy__(self,memo):". Syntax seems strange to me as someone still learning Python, but ok, memo, fine. This works in class VariableWithoutRE but still problem with class VariableWithRE! New error: "ValueError: Cannot process flags argument with a compiled pattern" – Marcus Jones Jun 10 '11 at 07:32
2

@user789215 The problem is that you are invoking the ``VariableWithoutRE`` constructor (__init__) with an already compiled regex (self.regexTarget), while the constructor expects a string. – ThomasH Jul 06 '11 at 12:10

score 8 · Answer 2 · answered Jul 08 '19 at 12:40

8

This can be worked around by patching copy module in pre-3.7 python:

import copy
import re 

copy._deepcopy_dispatch[type(re.compile(''))] = lambda r, _: r

o = re.compile('foo')
assert copy.deepcopy(o) == o

answered Jul 08 '19 at 12:40

Adam Sosnowski

1,126
9
7

Does this create a new instance? From my testing it looks like it but the code suggests that its the same instance? – Markus Ressel Oct 05 '19 at 01:30
1

@MarkusRessel: It does not -- as shown by `assert copy.deepcopy(o) is o`. How did you test it? – Adam Sosnowski Oct 07 '19 at 15:10
You are right, my testing was wrong. The `re.compile()` method has a built in cache, so even using something like `re.compile(r.pattern)` in the lambda will not create a new instance if the pattern (and flags) are the same. I wonder if this is how it is done in python 3.7 too. – Markus Ressel Oct 08 '19 at 20:14
It's effectively the same (i.e.: copy is implemented as identity) but implemented in native code for re module: https://github.com/python/cpython/commit/fdbd01151dbd5feea3e4c0316d102db3d2a2a412 – Adam Sosnowski Oct 09 '19 at 12:54

score 5 · Answer 3 · answered Oct 19 '18 at 03:28

This seems to be fixed in Python version 3.7+:

Compiled regular expression and match objects can now be copied using copy.copy() and copy.deepcopy(). (Contributed by Serhiy Storchaka in bpo-10076.)

as per: https://docs.python.org/3/whatsnew/3.7.html#re

Testing:

import re,copy

class C():
    def __init__(self):
       self.regex=re.compile('\d+')

myobj = C()    
foo = copy.deepcopy(myobj)
foo.regex == myobj.regex
# True

score 0 · Answer 4 · answered Sep 01 '18 at 15:24

The problem seems to be the compiled regex. deepcopy cannot handle them.

A minimal example gives me the same error:

import re,copy
class C():
    def __init__(self):
        self.regex=re.compile('\d+')

myobj = C()    
copy.deepcopy(myobj)

This throws the error: TypeError: cannot deepcopy this pattern object. I'm in python3.5.

JL Peyret · Answer 5 · 2019-12-03T01:38:17.480

IF your instances of this class (and its subclasses_) don't need deep-copy but are only causing problems because they are part of object graphs that do need deep-copy, then you could do this:

#might as well limit this hack to versions that need it...
if sys.version_info <= (3, 7):

    def __deepcopy__(self, *args, **kwargs):
        """ cheat here because regex can't be deepcopied"""
        return self

Now, you need to be careful re. the assumption I am making. If you start altering any instances of this class, you are at risk of having side effects appear because the deepcopy did not in fact take place. It's only worthwhile if you need deepcopy elsewhere and you are pretty sure you don't care about this class and deepcopy.

TypeError: cannot deepcopy this pattern object

5 Answers5

Linked