4

I'm trying to get rid of object pinned in shared memory using ray.put. Here is code sample:

import ray
<create obj>

for ...:
  obj_id = ray.put(obj)

  <do stuff with obj_id on ray Actors using ray.get(obj_id)>
  del obj_id

After this is finished, I look at ray dashboard and see that all obj_id are still in ray shared memory with reference type LOCAL_REFERENCE.

Official docs do not elaborate on whether there is any way of explicitly controlling object lifetime. As far as I understood, it basically suggests to wait until all memory is used, and then rely on ray to clean things up.

Question: how do I explicitly purge object from ray shared memory?

Note: I'm using Jupyter, can it be the case that this object is still alive due to this fact?

ptyshevs
  • 1,602
  • 11
  • 26
  • 1
    Have a look at `2. Objects pinned in memory` [here](https://docs.ray.io/en/master/memory-management.html#debugging-using-ray-memory). – Patol75 Aug 21 '20 at 04:57
  • 1
    Ok. So the answer is to look carefully through the code for hanging references. Once I remove all the references, ray will immediately clear the object from the shared memory. Do I understand it correctly? – ptyshevs Aug 21 '20 at 06:13
  • 1
    I am understanding the same, but I have not tried it. – Patol75 Aug 21 '20 at 06:15
  • 1
    I wonder if there is an easy way to see where all the references are located. Wouldn't it be convenient to have something like `ray.delete(obj_id)`, where ray will go over all references and delete those? – ptyshevs Aug 21 '20 at 06:22
  • 1
    In a sense, don't you just have to `del` references returned by `put` and `get`? – Patol75 Aug 21 '20 at 06:43
  • I have this reference from put inside a function, basically. After the function is finished I delete it, and look at Ray dashboard only to see that same obj_id with "LOCAL_REFERENCE" type. – ptyshevs Aug 21 '20 at 07:08
  • Is the actor somehow storing a reference to the argument? Could you provide a sample of the actor code? – Alex Aug 23 '20 at 04:38
  • Alex, actually yes, it is. But I thought all Actors will be automatically released, as they are also automatically dispatched by tune.run. Isn't it the case? – ptyshevs Aug 23 '20 at 07:27
  • Not sure if you are still stuck on this one, but maybe this [paragraph](https://docs.ray.io/en/master/actors.html#terminating-actors) can answer your last question? – Patol75 Sep 01 '20 at 11:31
  • 1
    @Patol75 great, that's what I was looking for! – ptyshevs Sep 01 '20 at 12:09
  • How did you solve it? I have the same issue. – Stefan Mar 31 '21 at 15:23
  • 1
    @Stefan Essentially, I ended up debugging my code using `ray memory`, you can read more here: https://docs.ray.io/en/master/memory-management.html – ptyshevs Mar 31 '21 at 16:14
  • Thanks, it appears an reference is being held by `ray` itself by these three files `%MyCondaEnv%\lib\site-packages\ray\serialization.py:object_ref_deserializer:45`, `%MyCondaEnv%\lib\site-packages\ray\function_manager.py:fetch_and_register_remote_function:180` and `%MyCondaEnv%\lib\site-packages\ray\import_thread.py:_run:87` Do you have a hint where to remove these references. It seems like a `ray` internal thing. – Stefan Mar 31 '21 at 16:56
  • 1
    @Stefan that can definitely also occur, from my experience. – ptyshevs Mar 31 '21 at 19:18

1 Answers1

2

The function ray.internal.internal_api.free() performs this function. I can't find any documentation on the Ray docs for this function but it has a good docstring you can find here which I've copy-pasted below.

Free a list of IDs from the in-process and plasma object stores.

This function is a low-level API which should be used in restricted
scenarios.

If local_only is false, the request will be send to all object stores.

This method will not return any value to indicate whether the deletion is
successful or not. This function is an instruction to the object store. If
some of the objects are in use, the object stores will delete them later
when the ref count is down to 0.

Examples:
    >>> x_id = f.remote()
    >>> ray.get(x_id)  # wait for x to be created first
    >>> free([x_id])  # unpin & delete x globally

Args:
    object_refs (List[ObjectRef]): List of object refs to delete.
    local_only (bool): Whether only deleting the list of objects in local
    object store or all object stores.
Oscar Knagg
  • 128
  • 1
  • 8