Why return anything but `self` from `__iadd__`?

Question

Python's documentation on the methods related to the in-place operators like += and *= (or, as it calls them, the augmented arithmetic assignments) has the following to say:

These methods should attempt to do the operation in-place (modifying self) and return the result (which could be, but does not have to be, self). If a specific method is not defined, the augmented assignment falls back to the normal methods.

I have two closely related questions:

• Why is it necessary to return anything from these methods if the documentation specifies that, if implemented, they should only be doing stuff in-place anyway? Why don't the augmented assignment operators simply not perform the redundant assignment in the case where __iadd__ is implemented?
• Under what circumstances would it ever make sense to return something other than self from an augmented assignment method?

A little experimentation reveals that Python's immutable types don't implement __iadd__ (which is consistent with the quoted documentation):

>>> x = 5
>>> x.__iadd__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute '__iadd__'

and the __iadd__ methods of its mutable types, of course, operate in-place and return self:

>>> list1 = []
>>> list2 = list1
>>> list1 += [1,2,3]
>>> list1 is list2
True

As such, I can't figure out what the ability to return things other than self from __iadd__ is for. It seems like it would be the wrong thing to do in absolutely all circumstances.

Answer 1

Why is it necessary to return anything from these methods if the documentation specifies that, if implemented, they should only be doing stuff in-place anyway? Why don't the augmented assignment operators simply not perform the redundant assignment in the case where __iadd__ is implemented?

One reason is to force them to be statements instead of expressions.

---

A bigger reason is that the assignment isn't always superfluous. In the case where the left-hand side is just a variable, sure, after mutating the object, re-binding that object to the name it was already bound to is usually not necessary.

But what about the case where the left-hand side is a more complicated assignment target? Remember that you can assign—and augmented-assign—to subscriptions, slicings, and attribute references, like a[1] += 2 or a.b -= 2. In that case, you're actually calling __setitem__ or __setattr__ on an object, not just binding a variable.

---

Also, it's worth noting that the "redundant assignment" isn't exactly an expensive operation. This isn't C++, where any assignment can end up calling a custom assignment operator on the value. (It may end up calling a custom setter operator on an object that the value is an element, subslice, or attribute of, and that could well be expensive… but in that case, it's not redundant, as explained above.)

---

And the last reason directly ties into your second question: You almost always want to return self from __ispam__, but almost always isn't always. And if __iadd__ ever didn't return self, the assignment would clearly be necessary.

---

Under what circumstances would it ever make sense to return something other than self from an augmented assignment method?

You've skimmed over an important related bit here:

These methods should attempt to do the operation in-place (modifying self)

Any case where they can't do the operation in-place, but can do something else, it will likely be reasonable to return something other than self.

Imagine an object that used a copy-on-write implementation, mutating in-place if it was the only copy, but making a new copy otherwise. You can't do that by not implementing __iadd__ and letting += fall back to __add__; you can only do it by implementing an __iadd__ that may make and return a copy instead of mutating and returning self. (You might do that for performance reasons, but it's also conceivable that you'd have an object with two different interfaces; the "high-level" interface looks immutable, and copies-on-write, while the "low-level" interface exposes the actual sharing.)

So, the first reason it's needed is to handle the non-in-place case.

---

But are there other reasons? Sure.

One reason is just for wrapping other languages or libraries where this is an important feature.

For example, in Objective C, lots of methods return a self which is usually but not always the same object that received the method call. That "not always" is how ObjC handles things like class clusters. In Python, there are better ways to do the same thing (even changing your class at runtime is usually better), but in ObjC, it's perfectly normal and idiomatic. (It's only used for init methods in Apple's current Framework, but it's a convention of their standard library that mutator methods added by NSMutableFoo always return void, just like the convention that mutator methods like list.sort always return None in Python, not part of the language.) So, if you wanted to wrap up the ObjC runtime in Python, how would you handle that?

You could put an extra proxy layer in front of everything, so your wrapper object can change up what ObjC object it's wrapping. But that means a whole lot of complicated delegation code (especially if you want to make ObjC reflection work back up through the wrapper into Python) and memory-management code, and a performance hit.

Instead, you could just have a generic thin wrapper. If you get back a different ObjC object than you started with, you return the wrapper around that thing instead of the wrapper around the one you started with. Trivial code, memory management is automatic, no performance cost. As long as the users of your wrapper always do a += b instead of a.__iadd__(b), they will see no difference.

I realize that "writing a PyObjC-style wrapper around a different ObjC framework library than Apple's Foundation` is not exactly an every-day use case… but you already knew that this is a feature you don't use every day, so what else would you expect?

A lazy network object proxy might do something similar—start with a tiny moniker object, swap that out for a full proxy object the first time you try to do something to it. You can probably think of other such examples. You will probably never write any of them… but if you had to, you could.

Answer 2

To answer your second question, you return something other than self when you are dealing with immutable objects.

Take this example:

>>> foo = tuple([1, 2, 3])
>>> foo
(1, 2, 3)

Now I try to mutate the object in place:

>>> foo += (4,)
>>> foo
(1, 2, 3, 4)

Can't believe that worked!...now we try to call its __iadd__ method:

>>> foo.__iadd__((5,))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'tuple' object has no attribute '__iadd__'

It doesn't have one! Strange...now you must wonder to yourself, what happened when we did foo += (4,)?

Well essentially what happened is that when such in-place modification is attempted on immutable objects, python uses the objects __add__ method to carry out the operation, i.e. a new object is created. So what actually happened was that when we did this:

>>> foo += (4,)

This happened:

>>> foo = foo + (4,)

See this

So when you attempt to use += or any other inplace modification on an immutable object, python will use the non-mutating equivalent of that operator and you will get back a new object. Another example:

>>> foo += (5,) + (6,)
>>> foo
(1, 2, 3, 4, 5, 6)

The above translates to:

foo = foo + (5,) + (6,)

Hope that answers your question