Diamond problem when using MixIns in Python

Question

Please consider the following code implementing a simple MixIn:

class Story(object):
    def __init__(self, name, content):  
     self.name = name
     self.content = content    

class StoryHTMLMixin(object):
    def render(self):
     return ("<html><title>%s</title>"
         "<body>%s</body></html>"
         % (self.name, self.content))

def MixIn(TargetClass, MixInClass):
    if MixInClass not in TargetClass.__bases__:
     TargetClass.__bases__ += (MixInClass,)

if __name__ == "__main__":
    my_story = Story("My Life", "<p>Is good.</p>")
    # plug-in the MixIn here
    MixIn(Story, StoryHTMLMixin)
    # now I can render the story as HTML
    print my_story.render()

Running main leads to the following error:

TypeError: Cannot create a consistent method resolution
order (MRO) for bases object, StoryHTMLMixin

The problem is that both Story and StoryHTMLMixin are derived from object, and the diamond problem arises.

The solution is simply to make StoryHTMLMixin an old-style class, i.e., remove the inheritance from object, thus, changing the definition of the class StoryHTMLMixin to:

class StoryHTMLMixin:
    def render(self):
     return ("<html><title>%s</title>"
         "<body>%s</body></html>"
         % (self.name, self.content))

leads to the following result when running main:

<html><title>My Life</title><body><p>Is good.</p></body></html>

I don't like having to use old style classes, so my question is:

Is this the correct way to handle this problem in Python, or is there a better way?

Edit:

I see that the class UserDict in the latest Python source defines a MixIn resorting to the old style classes (as presented in my example).

As recommended by all, I may resort to redefining the functionality that I want to attain (namely, the binding of methods at run time) without using MixIns. However, the point still remains - is this the only use case where messing with the MRO is unsolvable without resorting to reimplementation or falling back to old-style classes?

Switching back to old-style classes is never the correct way to handle anything in Python--they no longer even exist in Python 3. — Glenn Maynard, Dec 23 '10 at 00:10
Python already solves the diamond problem, by using C3 ordering on the base classes. You're mangling that order by hand, so it's your responsibility to maintain the MRO consistent: http://www.python.org/download/releases/2.3/mro/. Or you can just use subclassing. — lqc, Dec 23 '10 at 00:28

Jochen Ritzel · Answer 1 · 2010-12-23T00:42:52.887

5

Why don't you use mixins directly instead of hacking around in the mro?

class Story(object):
    def __init__(self, name, content):  
     self.name = name
     self.content = content    

class StoryHTMLMixin(object):
    def render(self):
     return ("<html><title>%s</title>"
         "<body>%s</body></html>"
         % (self.name, self.content))

class StoryHTML(Story, StoryHTMLMixin):
    pass


print StoryHTML('asd','asdf').render() # works just fine

If you really, really, really want to glue extra methods on a class, that is not a big problem. Well, except that it's evil and bad practice. Anyways, you can change a class at any time:

# first, the story
x = Story('asd','asdf')

# changes a class object
def stick_methods_to_class( cls, *methods):
    for m in methods:
        setattr(cls, m.__name__, m)

# a bare method to glue on
def render(self):
 return ("<html><title>%s</title>"
     "<body>%s</body></html>"
     % (self.name, self.content))

# change the class object
stick_methods_to_class(Story, render)

print x.render()

But in the end, the question remains: Why should classes suddenly grow extra methods? That's the stuff horror movies are made of ;-)

edited Dec 23 '10 at 00:42

answered Dec 23 '10 at 00:17

Jochen Ritzel

104,512
31
200
194

That is no longer a mixin; it's just a multiply-inherited subclass. Although I agree that it is clearer. – Katriel Dec 23 '10 at 00:19
1

I think a mixin is just a class without `__init__`, at least that's how the `SocketServer` module does it. What the OP does seems more like monkeypatching to me .. – Jochen Ritzel Dec 23 '10 at 00:20
@THC: so it is! My apologies. – Katriel Dec 23 '10 at 00:23
Creating another class is not a MixIn, because if the Story object is used elsewhere it will not see the MixIn methods. MixIn affects globally, a new class affects locally. – Escualo Dec 23 '10 at 00:28
1

Whatever you call it, modifying objects that already exist by editing the class is *evil*. Don't do that! – Glenn Maynard Dec 23 '10 at 00:33
@Glenn Maynard: yes, I agree this is evil, I was just curious about the error. – Escualo Dec 23 '10 at 00:39
@Glenn Maynard: Please explain why modifying existing objects is *evil* in general. I'm also thinking that in the case of class objects, that if it was, then why aren't they immutable... – martineau Dec 23 '10 at 04:18

Glenn Maynard · Accepted Answer · 2010-12-23T00:43:06.020

4

It's easier to see what's happening if we eliminate the __bases__ magic and write our the classes you're creating explicitly:

class StoryHTMLMixin(object):
    def render(self):
        return ("<html><title>%s</title>"
            "<body>%s</body></html>"
            % (self.name, self.content))

class Story(object, StoryHTMLMixin):
    def __init__(self, name, content):
        self.name = name
        self.content = content

That's the end result of what you're doing--or what it would be if it succeeded. It results in the same error.

Notice that this isn't actually diamond inheritance. That involves four classes, where two base classes each inherit a common fourth class; Python's multiple inheritance deals with that.

Here you have just three classes, leading to inheritance that looks like this:

StoryHTMLMixin <--- Story
            |   _____/
            |  |
            v  v
           object

Python doesn't know how to resolve this.

I don't know about a workaround for this. In principle the solution would be to remove object from the bases of Story at the same time you add StoryHTMLMixin to it, but that isn't allowed for somewhat opaque, internal reasons (TypeError: __bases__ assignment: 'StoryHTMLMixin' deallocator differs from 'object').

I've never found any practical, real-world use for modifying classes like this anyway. It just seems obfuscated and confusing--if you want a class that derives from those two classes, just create a class normally.

Ed:

Here's an approach that does something similar to yours, but without modifying classes in-place. Note how it returns a new class, deriving dynamically from the arguments of the function. This is much clearer--you can't inadvertently modify objects that are already instantiated, for example.

class Story(object):
    def __init__(self, name, content):
        self.name = name
        self.content = content

class StoryHTMLMixin(object):
    def render(self):
        return ("<html><title>%s</title>"
            "<body>%s</body></html>"
            % (self.name, self.content))

def MixIn(TargetClass, MixInClass, name=None):
    if name is None:
        name = "mixed_%s_with_%s" % (TargetClass.__name__, MixInClass.__name__)

    class CombinedClass(TargetClass, MixInClass):
        pass

    CombinedClass.__name__ = name
    return CombinedClass

if __name__ == "__main__":
    MixedStory = MixIn(Story, StoryHTMLMixin, "MixedStory")
    my_story = MixedStory("My Life", "<p>Is good.</p>")
    print my_story.render()

edited Dec 23 '10 at 00:43

answered Dec 23 '10 at 00:27

Glenn Maynard

55,829
10
121
131

I really like the approach you suggest, tough it does not affect the Story object globally (maybe this is a good thing?) – Escualo Dec 23 '10 at 00:36
That edit is how it should be if you really want to do it that way. This will not modify the class in place (all very well in Ruby but considered bad practice in Python). Plus, it also keeps things like `isinstance(my_story, Story)` working. @Glenn: you might like to accept an optional third parameter to override `CombinedClass.__name__`. – Chris Morgan Dec 23 '10 at 00:39
1

@Arrieta: Like I said elsewhere--modifying classes directly to globally affect previously-instantiated objects of that class is evil. Unless you're in a desperate monkey-patch situation (which we're all in now and then), avoid it like the plague. – Glenn Maynard Dec 23 '10 at 00:39
@Chris: Added. (An odd and somewhat inconsistent quirk that feels like a bug: you can't simply say `__name__ = value` within the class definition, probably due to the default being assigned after the class definition without checking to make sure it's not clobbering an explicitly-specified one.) – Glenn Maynard Dec 23 '10 at 00:44
2

Original question aside, this seems like a useful example to demonstrate how classes are simple objects like everything else; it takes a certain comfort level in the language to realize that this is possible. – Glenn Maynard Dec 23 '10 at 00:47
Thanks @Glenn - I will base my new implementation on the example you provide. – Escualo Dec 23 '10 at 01:17
@Glenn: Not being able to set `__name__` during the definition is what I would have expected; it'll be set as part of `type.__new__` process (or however the class is constructed), not before. – Chris Morgan Dec 23 '10 at 01:53
@Chris: If I set `a = 1` within the class body, the resulting `cls.a` attribute is 1. It's inconsistent that if I do the same with `__name__ = 'str'`, the same doesn't happen. (I'm sure this happens because it's hard to tell the difference between the `__name__` attribute being set explicitly within the body, versus being inherited.) – Glenn Maynard Dec 23 '10 at 02:34
`__name__` is magic. You wouldn't expect to inherit it, it's a per-class thing and it makes sense that it gets created after that step. I put it simply as a statement of fact that putting `__name__` in the class definition never occurred to me - it just felt wrong and I wouldn't have expected it to work. – Chris Morgan Dec 23 '10 at 02:39

Katriel · Answer 3 · 2010-12-23T00:35:57.160

2

EDIT: My bad, the bug is a separate issue (thanks @Glenn Maynard). The following hack still works, as long as your mixin inherits directly from object.

class Text(object): pass
class Story(Text):
    ....

However, I think that mixins are not the neatest solution to your problem. Both other solutions provided (a class decorator and an ordinary subclass) clearly mark the Story class as renderable, while your solution conceals this fact. That is, the existence of the render method in the other solutions is explicit while in your solution it is hidden. I think this will cause confusion later, especially if you come to rely more heavily on the mixin methodology.

Personally I like the class decorator.

edited Dec 23 '10 at 00:35

answered Dec 23 '10 at 00:17

Katriel

120,462
19
136
170

That's a different issue (and one I mentioned in my answer). The OP's issue isn't related to it--see the top of my answer and replace `object` with a subclass of `object` and the problem won't change. – Glenn Maynard Dec 23 '10 at 00:29
Thanks - yes, it seems related to the bug you link to. I agree - there are much better ways to implement dynamic binding of methods, but I was just surprised to find the specific diamond problem and wondered wether it could be solved cleanly. – Escualo Dec 23 '10 at 00:31
I feel it would be "less bad" if I use old style class for the MixIn than to create a new class which is meant to do nothing but fix the MRO. However, that is just my opinion and I have no reason to believe is the right approach in general. – Escualo Dec 23 '10 at 00:32
@Glenn: Yep, just realised that. Am still thinking about this, though; why is Python OK with `class Story(StoryHTMLMixin, object)` but not `class Story(object, StoryHTMLMixin)`? – Katriel Dec 23 '10 at 00:34
@Arrieta: I think that using old-style classes for anything is __bad, bad, bad__. They are thoroughly deprecated. – Katriel Dec 23 '10 at 00:37

score 2 · Answer 4 · answered Dec 23 '10 at 02:17

However, the point still remains - is this the only use case where messing with the MRO is unsolvable without resorting to reimplementation or falling back to old-style classes?

Others have suggested better solutions -- such as explicit construction of the desired class -- but to answer your edited question, it is possible to define the mixin without resorting to old-style classes:

class Base(object): pass

class Story(Base):
    def __init__(self, name, content):  
     self.name = name
     self.content = content    

class StoryHTMLMixin(Base):
    def render(self):
     return ("<html><title>%s</title>"
         "<body>%s</body></html>"
         % (self.name, self.content))

def MixIn(TargetClass, MixInClass):
    if MixInClass not in TargetClass.__bases__:
        TargetClass.__bases__ = (MixInClass,)+TargetClass.__bases__

if __name__ == "__main__":
    my_story = Story("My Life", "<p>Is good.</p>")
    # plug-in the MixIn here
    MixIn(Story, StoryHTMLMixin)
    # now I can render the story as HTML
    print my_story.render()

yields

<html><title>My Life</title><body><p>Is good.</p></body></html>

score 1 · Answer 5 · answered Dec 23 '10 at 00:05

1

You could try using a decorator to add the functionality instead:

def render(obj):
  return ("<html><title>%s</title>"
    "<body>%s</body></html>"
    % (obj.name, obj.content))

def renderable(cls):
  cls.render = render
  return cls

@renderable
class Story(object):
  ...

answered Dec 23 '10 at 00:05

Karl Knechtel

62,466
11
102
153

2

I agree that there are many ways to add method dynamically (some people are severely against MixIns, as I presented them here). However, the question remains, can you fix the diamond problem without resorting to old style classes? Thanks. – Escualo Dec 23 '10 at 00:14

score 1 · Answer 6 · answered Dec 23 '10 at 08:42

Aside from everything said by the previous answers (and being evil and a bad idea), your problem is:

Bases should be ordered the other way around

TargetClass.__bases__ = (MixInClass,) + TargetClass.__bases__
http://bugs.python.org/issue672115 - The easiest workaround is to inherit everything from a user defined class and not object, like
```
class MyBase(object): pass


class Story(MyBase):
     ...


class StoryHTMLMixin(MyBase):
     ...
```

Diamond problem when using MixIns in Python

6 Answers6