13

In this problem, I'm dealing with IPv6 network address spaces, so the length is 2^(128-subnet).

It appears that python (at least on this machine), will cope with up to a 64 bit signed number as the return value from __len__(). So len(IP('2001::/66')) works, but len(IP('2001::/65')) fails.

from IPy import IP
len(IP('2001::/64'))
Traceback (most recent call last):
  File "test.py", line 2, in <module>
      len(IP('2001::/64'))
OverflowError: long int too large to convert to int

The IPy library in question is at https://github.com/haypo/python-ipy.

Any suggestions on how to handle this, or hint that it might be a limitation I'm stuck with?

Jeff Ferland
  • 17,832
  • 7
  • 46
  • 76
  • +1 because I'm interested in this. On x64 Win7, using Python 2.6, `len(IP('2001::/64'))` gives me `TypeError: __len__() should return an int` whereas `IP('2001::/64').__len__()` gives me `18446744073709551616L`. What does `IP('2001::/64').__len__()` give you? – azhrei Mar 27 '13 at 02:31
  • @azhrei OverflowError instead of a TypeError in 2.7.3, but the same pattern with `.__len__()` working fine and `len()` raising the exception. – Jeff Ferland Mar 27 '13 at 02:34
  • Well, comments in IPy's source code suggest it's a python limitation you're stuck with. http://bugs.python.org/issue3729 seems to report the basic issue, and that it was found in 2.6 and 3.0. The bug is closed as fixed, but I can't see where it was fixed. Might be worth trying Python 3.3 ? – azhrei Mar 27 '13 at 03:08
  • @azhrei Python 3.2.3 gives me `OverflowError: cannot fit 'int' into an index-sized integer`. This might end up just being a documentation note to use `x.len()` and not `len(x)`. – Jeff Ferland Mar 27 '13 at 03:11

1 Answers1

13

The issue you're hitting is that Python's C API has a system-dependent limit on the lengths of containers. That is, the C function PyObject_Size returns a Py_ssize_t value, which is a signed version of the standard C size_t type. It's size is system dependent, but probably 32-bits on 32-bit systems and 64-bits on 64-bit systems.

The builtin len function uses PyObject_Size, so it has the same limitations. Here's its current implementation:

static PyObject *
builtin_len(PyObject *self, PyObject *v)
{
    Py_ssize_t res;

    res = PyObject_Size(v);
    if (res < 0 && PyErr_Occurred())
        return NULL;
    return PyInt_FromSsize_t(res);
}

You can work around this limitation by using the len method on the IP object, rather than calling the builtin len:

IP('2001::/64').len()

This is pure python, so it doesn't have any limitations on integer size.

Blckknght
  • 100,903
  • 11
  • 120
  • 169
  • Cool. I'll put it into IPy's documentation. – Jeff Ferland Mar 27 '13 at 03:19
  • Would it be worth reporting this as a bug? There is no reason why lengths and indices can't go over that size, especially when we have "virtual" sets like this one. – Lambda Fairy Mar 27 '13 at 04:43
  • @LambdaFairy I'm confused. Reporting it to who as a bug? – Jeff Ferland Mar 27 '13 at 04:56
  • 1
    Reporting it to the Python developers. According to the [official documentation](http://docs.python.org/2/reference/datamodel.html#object.__len__), `__len__` should return "an integer". A very large integer is still an integer. – Lambda Fairy Mar 27 '13 at 07:12