3

As someone who has been putting the Python C-API through its paces of late, I am curious about PyVarObject, which the documentation says is a subtype of PyObject used specifically for the definition of variable-length types – and though the caveat that:

This type does not often appear in the Python/C API

… there are an abundance of documented support structures, slot hooks, and other API minutiae that are dedicated to PyVarObject, to the point where I am genuinely curious if there are any Python-API-related problems (however specifically niche) for which it can notably ease the solution.

fish2000
  • 4,289
  • 2
  • 37
  • 76

1 Answers1

2

It's mostly for immutable container types (e.g. tuple and bytes). The idea is that the type can have an array at the end of it that contains all its data. When it is allocated (with PyObject_NewVar or similar) extra space is allocated for the (immutable) data it will contain, which simplifies memory management and presumably gains a bit of speed by only having to do a single memory allocation. Because the size never changes after construction it can take this shortcut without having to worry about invalidating references to it by reallocating the memory.

To confuse things slightly list is also a PyVarObject despite being mutable. However, it has an item size of 0 and so no extra space is actually allocated at the end of the object. Instead it stores a pointer to an array of PyObject*, allocated separately (which can be reallocated if the list grows without invalidating references to the list). Presumably it was made a PyVarObject just to have an ob_size field.

DavidW
  • 29,336
  • 6
  • 55
  • 86
  • Aha, interesting – so would you say that it’s more CPython implementation detail than useful `PyObject` subtype? – fish2000 May 16 '16 at 23:45
  • 1
    Somewhere between. It certainly is a CPython implementation detail. But it does provide a small benefit for certain container types, so if you were implementing something like a (non-resizable) numpy array, you might well chose to use it yourself. I don't think anything it would be a huge loss not to have it though. – DavidW May 17 '16 at 17:50
  • If I understand you correctly, the storage for a `PyVarObject* op` is everything after the `PyObject_HEAD` stuff in the memory region from the object pointer to e.g. `op + op->ob_size`, yes? … which OK wow, I find that personally very funny as I am currently working on exactly that sort of thing – I’m writing a Python extension in C++ to replace PIL for simple image I/O (backed by immutable arrays). It might be good to use a `PyVarObject` for implementing, like, a hyperslab selection in a Pythonic way (as an array index or somesuch). Neat! – fish2000 May 17 '16 at 18:11
  • 1
    Yes - that's substantially it. The extra space allocated is `op->ob_size` times the `tp_itemsize` field of the `TypeObject`. Just remember that you can't resize it after you've allocated it. – DavidW May 17 '16 at 19:57
  • Awesome. That works for me… I like to do allocations like Louis Khan did concrete, boundary-aligned and fearsomely immutable. Thanks for the cogent breakdown @DavidW – I did not expect `PyVarObject` to have such usefulness on offer. – fish2000 May 17 '16 at 20:23