3

I was reading this question: Cannot return int array because I ran into the same problem.

It seems that data structures (because C can obviously return a locally declared variable) declared locally within a function cannot be returned, in this case an array.

However Python doesn't suffer from the same problem; as far as I can remember, it's possible to declare an array within a function and to return that array without having to pass it as an argument.

What is the difference "under the hood"? Is Python using pointers implicitly (using malloc within the function)?

Community
  • 1
  • 1
jeremy radcliff
  • 1,049
  • 3
  • 11
  • 27
  • 2
    Everything in Python is a pointer. In CPython, they're literally `PyObject*` and are refcounted. There's no way to allocate an object on the stack (which is the problem in C - things on the stack will disappear after a return, so if you try to return a pointer to your local stack frame, it'll become a dangling pointer). – Kevin Jan 31 '17 at 01:37
  • @Kevin, thanks for the explanation. – jeremy radcliff Jan 31 '17 at 01:39
  • Up voted because this is actually a good question. I don't know who down voted but you should be ashamed of yourself. – the_constant Jan 31 '17 at 01:39
  • Python is a high-level language with it's own memory management and garbage collection, and all objects are allocated dynamically. C is a low-level language in which you have to manage memory yourself, and locals are usually allocated on the return stack. – Lee Daniel Crocker Jan 31 '17 at 01:40
  • @Vincenzzzochi, thank you...I didn't understand the downvote either, seemed a little brutal. – jeremy radcliff Jan 31 '17 at 01:40
  • @LeeDanielCrocker, I guess that makes sense, it's just surprising (from the perspective of my knowledge base) that *all* objects are dynamically allocated. – jeremy radcliff Jan 31 '17 at 01:41
  • I thought it was I on a mis-click. It wasn't ... but it's still a decent question. However, please give an example: since Python doesn't have a specific **array** type, you need to clarify what you mean. An example would give us a particular on which to focus. – Prune Jan 31 '17 at 01:42
  • @Prune, you're right I should've been more precise; I actually meant a list in Python but automatically said an array because I discovered not too long ago that they are implemented as arrays under the hood. – jeremy radcliff Jan 31 '17 at 01:44
  • 1
    "Why can Python functions return locally declared arrays but C can't?" Because Python isn't C? Why should restrictions on C functions have any relevance to the semantics of Python functions? – John Coleman Jan 31 '17 at 01:51
  • @JohnColeman, I thought it'd be obvious I was asking about the difference in "under the hood" behavior between the two languages, but good job interpreting what I said literally so that you could slide in a passive aggressive comment. Obviously since Python is built on C, asking about relative behvaior between the two languages might be interesting, no? – jeremy radcliff Jan 31 '17 at 02:02
  • It is a good question, but would have been clearer if you used the word "how" rather than "why". – John Coleman Jan 31 '17 at 02:15
  • 1
    @jeremyradcliff I'm not sure responding in kind is particularly helpful either, though. YMMV. – Dave Newton Jan 31 '17 at 02:17
  • @JohnColeman, you're right and I apologize for my reaction, definitely over-the-top. – jeremy radcliff Jan 31 '17 at 21:16
  • @jeremyradcliff No problem. I have a tendency to be too pedantic and you were perhaps a little on the defensive side after receiving a couple unjustified downvotes (not from me, by the way). – John Coleman Jan 31 '17 at 22:07

2 Answers2

5

For the record, Python's built-in mutable sequence type is called a list, not an array, but it behaves similarly (it's just dynamically resizable, like C++'s std::vector).

In any event, you're correct that all Python objects are implicitly dynamically allocated; only the references (roughly, pointers) to them are on the "stack" (that said, the Python interpreter stack and the C level stack are not the same thing to start with). Comparable C code would dynamically allocate the array and return a pointer to it (with the caller freeing it when done; different Python interpreters handle this differently, but the list would be garbage collected when no longer referenced in one way or another).

Python has no real concept of "stack arrays" (it always returns a single object, though that object could be a tuple to simulate multiple return values), so returns are always ultimately a single "pointer" value (the reference to the returned object).

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
  • so in Python, when you index into a data structure, are you implicitly dereferencing a pointer? – jeremy radcliff Jan 31 '17 at 01:56
  • @jeremyradcliff: Usually (many) more than one pointer dereference is involved. For example, in CPython (the reference interpreter) if you have a `list` `mylist`, and do `mylist[2]`, it has to: 1. Dereference the pointer to `mylist` 2. Lookup the type of `mylist` (a pointer in the `PyObject` structure) and look for implementations of the C equivalent to the `__getitem__` special method, and call the one it finds 3. Dereference the pointer to the object representing `2` to pull out the C level value 4. Dereference the pointer to the array of object pointers, at offset 2 – ShadowRanger Jan 31 '17 at 02:03
  • I probably missed a few additional dereferences here too. All along the way, there are a ton of implicit pointer traversals involved, to maintain reference counts (the value being looked up has its reference count incremented before it's passed back, and even if you never even store the result, that increment and decrement have to happen), perform type checking and dynamic dispatch, etc. Python is much higher level; getting hung up on exact operations is usually a bad idea. – ShadowRanger Jan 31 '17 at 02:05
  • Thank you for explaining all of that; even if I don't understand all of it it's interesting to realize how much stuff is hidden in higher-level languages. I had no idea it was to that extent. – jeremy radcliff Jan 31 '17 at 02:10
1

It seems that data structures (because C can obviously return a locally declared variable) declared locally within a function cannot be returned, in this case an array.

You already have a good Python answer; I wanted to look at the C side a little more closely.

Yes, a C function returns a value. That value may be primitive C type, or a struct or union type. Or, it may be a pointer type.

The C language syntax makes arrays and pointers seem very similar, which makes arrays special. Because the name of the array is the same as the address of the first element, it can't be something else. In particular, an array name does not refer to the whole array (except in the case of the sizeof operator). Because any other use of an array name refers to the address of the first element, attempting to return an array results in returning only that address.

Because it's a C function, that address is returned by value: namely, a value of a pointer type. So, when we say, 

char *s = strdup("hello");

s is a pointer type whose value is not "hello", but the value of address of the first element of the array that strdup allocates.

Python doesn't suffer from the same problem

When Y is a property of X, Y is a problem only if that property is, in the eyes of the beholder, undesirable. You can be sure the way C treats arrays is not accidental, and is often convenient.

James K. Lowden
  • 7,574
  • 1
  • 16
  • 31