Ruby code memory leak in loop

Question

Below code is having memory leak. It's running under ruby 2.1.1. I am not able to find the actual leak.

q = Queue.new("test")
while true do
  m = q.dequeue
  body = JSON.parse(m.body)
  user_id = body["Records"][0]
  user = V2::User.find(user_id)
  post = V2::Post.find(post_id)  
end

After few hours of run I added GC.start but its not solving the problem

q = Queue.new("test")
while true do
  m = q.dequeue
  body = JSON.parse(m.body)
  user_id = body["Records"][0]
  user = V2::User.find(user_id)
  post = V2::Post.find(post_id) 
  GC.start 
end

I don't know how to find the actual memory leak.

How is `Queue` defined? What does `Queue#dequeue` look like (I noticed `dequeue` is not defined for the `Queue` that comes with Ruby)? Who enqueues objects into the queue? How many objects are in the queue on average? — spickermann, Jun 22 '16 at 06:22

Dbz · Accepted Answer · 2016-06-22T07:10:22.620

2

Try removing the lines from the bottom up, and seeing if the memory leak persists. It's possible that the Memory leak is coming from the find method, or possibly the JSON.parse (extremely unlikely), or the custom Queue data structure. If the memory leak is still there after removing all of the lines, it is likely coming from the worker itself and/or the program running the workers.

q = Queue.new("test")
while true do
  m = q.dequeue # Finally remove this and stub the while true with a sleep or something
  body = JSON.parse(m.body) # Then remove these two lines
  user_id = body["Records"][0]
  user = V2::User.find(user_id) # Remove the bottom two lines first
  post = V2::Post.find(post_id)
end

edited Jun 22 '16 at 07:10

answered Jun 22 '16 at 06:22

Dbz

2,721
4
35
53

At a first glance I was thinking the same. That loop would lead to a high CPU usage for sure, but why should it lead to a high memory usage? – spickermann Jun 22 '16 at 06:35
@Dbz Actually its a worker. It will run for every sec. It will Liston the queue. if any message is pushed to queue it will take the message from queue and do the job. – SaravanaKumAr Jun 22 '16 at 06:47
Yup. You guys are definitely correct. I've updated my answer with debugging tips. Make sure to post the results after trying this. – Dbz Jun 22 '16 at 07:11
2

@SaravanaKumAr How on the Earth the answer consisting of “probably” and “likely” got approved as correct? Did you check this assumption? You claimed it took hours for the leak to notice? Please avoid marking answers that **were not proven to work for you**. – Aleksei Matiushkin Jun 22 '16 at 07:19
@mudasobwa I think you linked to an excellent and informative stack overflow answer. His question stated he did not know **how** to find the memory leak. If my answer including 'probably' and 'likely' was able to help him find the memory leak, **and it worked for him**, then it doesn't seem unreasonable to up vote and/or mark it accepted. That being said, your answer on **why** and what he can do is equally important. – Dbz Jun 22 '16 at 16:24

score 0 · Answer 2 · edited May 23 '17 at 12:33

0

I bet the problem is with introduced local variables (sic!). Get rid of user_id and post_id and it’ll likely stop leaking:

# user_id = body["Records"][0]
# user = V2::User.find(user_id)
user = V2::User.find(body["Records"][0]) # sic!

The reason is how Ruby stores objects in RValues.

edited May 23 '17 at 12:33

Community

1
1

answered Jun 22 '16 at 07:16

Aleksei Matiushkin

119,336
10
100
160

After each GC.start the local variables memory will be freed rite? – SaravanaKumAr Jun 22 '16 at 07:24
For example am running a loop 1000 times. Inside that loop each time am just assigning user_id = some_value. Also each loop am running GC.start too. In that case when the memory will be freed?. What's your view on this. while i < 1000 i++ user_id = i GC.start end – SaravanaKumAr Jun 22 '16 at 07:29
I have linked a detailed description and asked you to read it twice. `GC` **does not release heaps**. Let me repeat it again: `GC` **does not release heaps**. It does not matter how many times you have it run, `GC` **does not release heaps**. So, on each subsequent iteration of the loop, each `user_id`, that obviously fits 40 bytes, will allocate a new heap that **will not be released back to OS**. – Aleksei Matiushkin Jun 22 '16 at 07:33
I understood your answer. What am asking is when that heap will released back to OS? – SaravanaKumAr Jun 22 '16 at 07:39
Never. **Please read the answer I linked**, it contains a detailed explanation. – Aleksei Matiushkin Jun 22 '16 at 07:40
Never. Your answer doesn't clarify when the heap memory released back to OS? – SaravanaKumAr Jun 22 '16 at 07:50
Are you bullying at me? The heap memory is **never released back to OS**. – Aleksei Matiushkin Jun 22 '16 at 07:54
Heap memory reserved by Ruby is release when the process terminates. But you are right @SaravanaKumAr - the memory generally should go back into the available pool for Ruby if it is garbage collected. Defining when that happens is beyond my expertise, but involves RValues (as mentioned in this answer), and references and scopes, and GC settings, etc. – mltsy Apr 10 '20 at 22:35

Ruby code memory leak in loop

2 Answers2