How to return array from a PHP extension, without copying it in memory?

Question

I'm developing a PHP-extension, where an object method needs to return an array zval.

The method looks like:

ZEND_METHOD(myObject, myMethod)
{
    zval **myArrayProperty;
    if (zend_hash_find(Z_OBJPROP_P(getThis()), "myArrayProperty", sizeof("myArrayProperty"), (void **) &myArrayProperty) == FAILURE) {
        RETURN_FALSE;
    }
    RETURN_ZVAL(*myArrayProperty, 1, 0);
}

The code works fine and does the expected thing - it returns object's myArrayProperty. However, I'd like to optimize the process.

myArrayProperty stores an array, which can be quite big. And the RETURN_ZVAL() macro duplicates that array in order to return the value. The duplication process takes good amount of time to acquire memory and copy all the array values. At the same time, the returned array is usually used for read-only operations. So a nice optimization would be to use PHP's mechanism with reference counting and do not duplicate myArrayProperty contents. Rather I'd increase refcount of myArrayProperty and just return pointer to it. This is the same tactic as usually used, when working with variables in a PHP extension.

However, seems, there is no way to do it - you have to duplicate value in order to return it from a PHP extension function. Changing function signature to return value by reference, is not an option, because it links the property and returned value - i.e. changing returned value later, changes the property as well. That is not an acceptable behavior.

The inability to engage reference counting looks strange, because same code in PHP:

function myMethod() {
{
    return $this->myArrayProperty;
}

is optimized by the reference counting mechanism. That's why I'm asking this question at StackOverflow in case I missed something.

So, is there a way to return an array from a function in PHP extension, without copying the array in memory?

NikiC · Answer 1 · 2013-08-31T15:52:56.927

7

If your function returns by-value this is only possible as of PHP 5.6 (current master) using the RETURN_ZVAL_FAST macro:

RETURN_ZVAL_FAST(*myArrayProperty);

If your function returns by-reference (return_reference=1 in the arginfo) you can return using the following code:

zval_ptr_dtor(&return_value);
SEPARATE_ZVAL_TO_MAKE_IS_REF(myArrayProperty);
Z_ADDREF_PP(myArrayProperty);
*return_value_ptr = *myArrayProperty;

If your function returns by-value and you're on PHP 5.5 or older you can still optimize the refcount=1 case:

if (Z_REFCOUNT_PP(myArrayProperty) == 1) {
    RETVAL_ZVAL(*myArrayProperty, 0, 1);
    Z_ADDREF_P(return_value);
    *myArrayProperty = return_value;
} else {
    RETVAL_ZVAL(*myArrayProperty, 1, 0);
}

edited Aug 31 '13 at 15:52

answered Aug 01 '13 at 11:05

NikiC

100,734
37
191
225

Well, as it is said in the description - the function doesn't return value by reference. So it is not a solution, unfortunately. – Andrey Tserkus Aug 01 '13 at 16:36
Sorry, I missed that. In this case what you want is not possible. – NikiC Aug 01 '13 at 17:17
Though I don't see an immediate reason why we couldn't pass in return_value_ptr even without ACC_RETURN_REFERENCE (apart from the fact that this would allow you to return an is_ref=1 zval from a non-ref function). You might want to ask on internals@ about this. – NikiC Aug 01 '13 at 17:21
It doesn't work, because return_value_ptr is initialized by engine only when return by reference is declared for a function/method. – Andrey Tserkus Aug 02 '13 at 15:08
@AndreyTserkus I'm aware of that :) Just saying that this is something we might want to change. It's not clear to me why we couldn't just always set return_value_ptr. – NikiC Aug 02 '13 at 16:37
Got it, @NikiC so it were just the thoughts on possible future improvements, not the solution. – Andrey Tserkus Aug 02 '13 at 16:47
@NikiC has just pushed the changes to PHP to address this issue, so it should be included in the next PHP 5.5.3 release. – TerryE Aug 31 '13 at 14:13
@TerryE I applied the patch only to master, so this is 5.6 only. I'm not sure this can be applied to 5.5 because it changes the binary API (it changes the expectations towards implementers of zend_execute_internal). – NikiC Aug 31 '13 at 15:55
@NikiC, thanks for the clarification / correction, which the readers here need. But I've already regressed it into my local 5.5.3 so I can benchmark it :-) – TerryE Aug 31 '13 at 17:36
@TerryE I'm pretty sure you will not get any measurable difference for "normal" code. The only thing that *might* have some impact is the change to `__call`. But that only if it's used a lot to return large arrays... – NikiC Aug 31 '13 at 18:54
@NikiC, Not so sure that's the case, but here isn't the place to have this chat. Let me post back on the Internals DL when I've collected the stats _if_ there is anything material to report or I can use the PHP chatroom. TTFN (Ta-Ta for now) – TerryE Aug 31 '13 at 22:23

jgmjgm · Answer 2 · 2015-10-27T13:34:07.263

I don't have access to PHP < 5.6 but I think the problem is that the value is copied only. To be absolutely sure you should search the code for the defines in question.

That means you might be able to try:

 zval *arr;
 MAKE_STD_ZVAL(arr);
 array_init(arr);
 // Do things to the array.
 RETVAL_ZVAL(arr, 0, 0);
 efree(arr);

This is dangerous if used unwisely. If used with your own temporary containers I don't know of any problems.

You can also probably work on return value directly which might be a better approach. You would likely initialise it and pass it around as a pointer at the start.

You can wrap your return result like this. You can also experiment with references.

JimiDini · Answer 3 · 2013-07-24T21:44:16.677

-1

It's been awhile, since I coded something like this…

So, what I do in code below: 1). explicitly increasing refcounter 2). returning zval without copying it

ZEND_METHOD(myObject, myMethod)
{
    zval **myArrayProperty;

    if (zend_hash_find(Z_OBJPROP_P(getThis()), "myArrayProperty", sizeof("myArrayProperty"), (void **) &myArrayProperty) == FAILURE) {
        RETURN_FALSE;
    }

    Z_ADDREF_PP(myArrayProperty);
    RETURN_ZVAL(*myArrayProperty, 0, 0);
}

edited Jul 24 '13 at 21:44

answered Jul 24 '13 at 21:05

JimiDini

2,039
12
19

But isn't it going to lead to a memory leak or segfault (whichever comes first)? Memory leak will happen, when all the references to the property are cleared, but the memory, occupied by its zval container cannot be freed, because the refcount would still remain 1. Segfault will happen, when the returned value is disposed, thus its zval container is cleared together with the array (shared HashTable, which is referenced both from that container and from property zval container), so using property later will lead to unpredictable, but definitely wrong effects. – Andrey Tserkus Jul 25 '13 at 07:20
Property references zval — that's refcount=1. this code increases refcount, as zval is returned and will be referenced by both object and the caller. if the object doesn't need property it will decrease refcount, so it will be owned only by caller. so, this code looks sane to me. but, again, all of this is strictly theoretical — I don't even have sources of PHP unpacked right now – JimiDini Jul 25 '13 at 09:06
unfortunately, as predicted, the code doesn't work - confirmed in practice: http://pastebin.com/FRfaJZvL . The issue is as said above: the object and the caller reference different memory locations. – Andrey Tserkus Jul 26 '13 at 02:40

How to return array from a PHP extension, without copying it in memory?

3 Answers3