1

This is a follow up of my previous question : Enum comparison become False after reloading module

Ultimately, I would like to be able to pickle my enum.

Let's start from myenum.py again :

# myenum.py
import enum

class MyEnum(enum.Enum):
    ONE = 1
    TWO = 2

I again import this file in my script. I create a variable a an instance of MyEnum, pickles it and load it into a variable b. It works fine and both variables are equal.

Now, I reload my file. I try to pickle a but the following error occurs :

Traceback (most recent call last):
  File "f:/python_test/test.py", line 8, in <module>
    b = pickle.loads(pickle.dumps(a))
_pickle.PicklingError: Can't pickle <enum 'MyEnum'>: it's not the same object as myenum.MyEnum

I believe this is because the IDs of the enum changed, so in pickle eyes, a is indeed not the same object.

Note that it is not a solution for me to redefine each existing enum variable each time a file is reimported.

Here is the code to reproduce the issue :

# test.py
import importlib, myenum, pickle

if __name__=='__main__':
    a = myenum.MyEnum.ONE
    b = pickle.loads(pickle.dumps(a))
    print(b == a) # is True
    importlib.reload(globals()["myenum"])
    b = pickle.loads(pickle.dumps(a)) # Error
    print(b == a)
bad_coder
  • 11,289
  • 20
  • 44
  • 72
Kusefiru
  • 130
  • 7
  • May I ask what you are trying to achieve be pickling an `Enum`? – DeepSpace Mar 03 '21 at 15:59
  • 2
    "Now, I reload my file." - congrats, you've encountered one of the reasons why module reloading is a terrible idea you should never rely on. – user2357112 Mar 03 '21 at 16:00
  • 1
    This has nothing to do with IDs changing. Object IDs cannot change. Reloading the module has created an entirely new enum class with entirely new instances. – user2357112 Mar 03 '21 at 16:01
  • @DeepSpace The end goal is not to pickle the enum by itself, but an object that may have one or multiple enum attributes. I simplified the example for demonstration purpose. – Kusefiru Mar 03 '21 at 16:06
  • @user2357112supportsMonica indeed the wording was wrong, but you got the idea. What would be an alternative to reloading the module ? – Kusefiru Mar 03 '21 at 16:07
  • 1
    @Kusefiru: That depends on why you're reloading it, but restarting Python is often appropriate. – user2357112 Mar 03 '21 at 16:38
  • This is unfortunately something I would like to avoid (which is why I reload the modules in the first place). We are working on large datasets and restarting the software would mean reloading those datasets each time the user want to change something in the imported module. – Kusefiru Mar 03 '21 at 16:47
  • 2
    Side-note: Why `importlib.reload(globals()["myenum"])` instead of just `importlib.reload(myenum)`? It feels like you don't truly understand what the code is doing, and you're applying weird hacks to it that you think will help, but change nothing. – ShadowRanger Mar 11 '21 at 01:51

2 Answers2

3

Short answer: Stop using reload. It's a hack for use during active development, not for production use.

If this is just for active development in an interactive session, move the definition of the enum somewhere aside from the module you're actively editing and reloading, so it doesn't get caught up in the reloads. Otherwise, you're stuck; pickle is accurately describing the error. After the reload, a (the old myenum.MyEnum.ONE) is entirely unrelated to myenum.MyEnum.ONE (for that matter, the class of a is unrelated to myenum.MyEnum). Sure, it may have been defined in logically the same way, but both the class and instance were redefined from scratch, and given a key selling point of enum (from the docs: "Within an enumeration, the members can be compared by identity"), the fact that they aren't the same object means they aren't equal. You're asking pickle to take an object of a class pickle can't find, and serialize it such that you can reproduce it, but that class is gone; pickle can't reproduce objects from it because it can't look the class up in any reliable place in your code (if it tried to import your module fresh, it could make the new ONE, but not your old ONE that's not equal to the current ONE).

You can't work around that without invoking the most terrible and brittle of hacks. So don't do it; find a way to avoid reloading things that must be pickled.

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
  • 2
    This isnt very helpful, because exactly the same happens if you are trying to use multiprocessing.Pool, it will pickle the arguments and pass them the function using the serialized object format, but it will break if the argument is an enum or contains enums inside of it. ```` File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pickle.py", line 1075, in save_global raise PicklingError( _pickle.PicklingError: Can't pickle : it's not the same object as events.EventType ```` There is no reload involved and the result is exactly the same. – Lucas Coppio Mar 29 '22 at 17:44
  • @LucasScoppio: I'm essentially 100% certain you misused `enum`; you likely did one of 1) Used [the functional API](https://docs.python.org/3/library/enum.html#functional-api) and assigned to a name different from the string you passed as the name, 2) Failed to define it at the top level of the module, or 3) Reassigned the name multiple times in the course of the module, and tried to pickle objects defined at one point after redefining the type. [`enum`s pickle just fine](https://docs.python.org/3/library/enum.html#pickling), you're just misusing them. – ShadowRanger Mar 29 '22 at 21:43
  • 1
    @LucasScoppio: In the future, I'd suggest not down-voting until you're actually sure there is something wrong with the answer; I guarantee you properly used `enum`s work just fine with multiprocessing (in `'fork'` or `'spawn'` mode, using the functional or class-oriented method of definition; I've tested them all locally; I'd provide a TIO link, but TIO doesn't allow you to use `multiprocessing`). – ShadowRanger Mar 29 '22 at 21:45
0

I was facing the same issue with pickling a class when the module was reloaded somewhere in between. Fortunately, it's quite easy to work around this using dill instead of pickle.

amks1
  • 43
  • 7