'Nesting too deep' error while retrieving JSON using HTTParty

Question

I am writing a very simple proxy wrapper for the reddit api so I can make cross domain JSONP requests (reddit does not allow JSONP calls to my knowledge).

I am using the HTTParty gem to make the call out and retrieve the response, which I immediately render out as json.

I am retrieving the subreddit json perfectly fine so far with this as my route:

  match "r/:name" => "api#subreddit"

... and this as my controller action:

def subreddit
    render :json => HTTParty.get("http://www.reddit.com/r/" + params[:name] + "/.json"), :callback => params[:callback]
end

As an example, the JSON that comes back is similar to this: http://www.reddit.com/r/pics/.json

My problem is when I attempt to grab the comments for a particular thread.

My route is as so:

match "r/:name/comments/:id" => "api#comments"

... and my controller action is this:

  def comments
    render :json => HTTParty.get("http://www.reddit.com/r/" + params[:name] + "/comments/" + params[:id] + "/.json"), :callback => params[:callback]    
  end

The JSON that I want back can be seen here: http://www.reddit.com/r/pics/comments/rhzgc/.json

When I call this action, however, I receive the following error message:

MultiJson::DecodeError in ApiController#comments
Nesting of 20 is too deep

The full stack trace follows:

json (1.6.6) lib/json/common.rb:148:in `parse'
json (1.6.6) lib/json/common.rb:148:in `parse'
multi_json (1.2.0) lib/multi_json/engines/json_common.rb:9:in `decode'
multi_json (1.2.0) lib/multi_json.rb:81:in `decode'
httparty (0.8.1) lib/httparty/parser.rb:116:in `json'
httparty (0.8.1) lib/httparty/parser.rb:136:in `parse_supported_format'
httparty (0.8.1) lib/httparty/parser.rb:103:in `parse'
httparty (0.8.1) lib/httparty/parser.rb:66:in `call'
httparty (0.8.1) lib/httparty/request.rb:222:in `parse_response'
httparty (0.8.1) lib/httparty/request.rb:192:in `handle_response'
httparty (0.8.1) lib/httparty/request.rb:75:in `perform'
httparty (0.8.1) lib/httparty.rb:391:in `perform_request'
httparty (0.8.1) lib/httparty.rb:343:in `get'
httparty (0.8.1) lib/httparty.rb:423:in `get'
app/controllers/api_controller.rb:12:in `comments'
actionpack (3.1.1) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
actionpack (3.1.1) lib/abstract_controller/base.rb:167:in `process_action'
actionpack (3.1.1) lib/action_controller/metal/rendering.rb:10:in `process_action'
actionpack (3.1.1) lib/abstract_controller/callbacks.rb:18:in `block in process_action'
activesupport (3.1.1) lib/active_support/callbacks.rb:416:in `_run__2872109728488784816__process_action__2261783022290592822__callbacks'
activesupport (3.1.1) lib/active_support/callbacks.rb:386:in `_run_process_action_callbacks'
activesupport (3.1.1) lib/active_support/callbacks.rb:81:in `run_callbacks'
actionpack (3.1.1) lib/abstract_controller/callbacks.rb:17:in `process_action'
actionpack (3.1.1) lib/action_controller/metal/rescue.rb:17:in `process_action'
actionpack (3.1.1) lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
activesupport (3.1.1) lib/active_support/notifications.rb:53:in `block in instrument'
activesupport (3.1.1) lib/active_support/notifications/instrumenter.rb:21:in `instrument'
activesupport (3.1.1) lib/active_support/notifications.rb:53:in `instrument'
actionpack (3.1.1) lib/action_controller/metal/instrumentation.rb:29:in `process_action'
actionpack (3.1.1) lib/action_controller/metal/params_wrapper.rb:201:in `process_action'
activerecord (3.1.1) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
actionpack (3.1.1) lib/abstract_controller/base.rb:121:in `process'
actionpack (3.1.1) lib/abstract_controller/rendering.rb:45:in `process'
actionpack (3.1.1) lib/action_controller/metal.rb:193:in `dispatch'
actionpack (3.1.1) lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
actionpack (3.1.1) lib/action_controller/metal.rb:236:in `block in action'
actionpack (3.1.1) lib/action_dispatch/routing/route_set.rb:65:in `call'
actionpack (3.1.1) lib/action_dispatch/routing/route_set.rb:65:in `dispatch'
actionpack (3.1.1) lib/action_dispatch/routing/route_set.rb:29:in `call'
rack-mount (0.8.3) lib/rack/mount/route_set.rb:152:in `block in call'
rack-mount (0.8.3) lib/rack/mount/code_generation.rb:96:in `block in recognize'
rack-mount (0.8.3) lib/rack/mount/code_generation.rb:89:in `optimized_each'
rack-mount (0.8.3) lib/rack/mount/code_generation.rb:95:in `recognize'
rack-mount (0.8.3) lib/rack/mount/route_set.rb:141:in `call'
actionpack (3.1.1) lib/action_dispatch/routing/route_set.rb:532:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
rack (1.3.6) lib/rack/etag.rb:23:in `call'
rack (1.3.6) lib/rack/conditionalget.rb:25:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/head.rb:14:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/params_parser.rb:21:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/flash.rb:243:in `call'
rack (1.3.6) lib/rack/session/abstract/id.rb:195:in `context'
rack (1.3.6) lib/rack/session/abstract/id.rb:190:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/cookies.rb:331:in `call'
activerecord (3.1.1) lib/active_record/query_cache.rb:62:in `call'
activerecord (3.1.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:477:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (3.1.1) lib/active_support/callbacks.rb:392:in `_run_call_callbacks'
activesupport (3.1.1) lib/active_support/callbacks.rb:81:in `run_callbacks'
actionpack (3.1.1) lib/action_dispatch/middleware/callbacks.rb:28:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/reloader.rb:68:in `call'
rack (1.3.6) lib/rack/sendfile.rb:101:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/remote_ip.rb:48:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/show_exceptions.rb:47:in `call'
railties (3.1.1) lib/rails/rack/logger.rb:13:in `call'
rack (1.3.6) lib/rack/methodoverride.rb:24:in `call'
rack (1.3.6) lib/rack/runtime.rb:17:in `call'
activesupport (3.1.1) lib/active_support/cache/strategy/local_cache.rb:72:in `call'
rack (1.3.6) lib/rack/lock.rb:15:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/static.rb:53:in `call'
railties (3.1.1) lib/rails/engine.rb:456:in `call'
railties (3.1.1) lib/rails/rack/content_length.rb:16:in `call'
railties (3.1.1) lib/rails/rack/log_tailer.rb:14:in `call'
rack (1.3.6) lib/rack/handler/webrick.rb:59:in `service'
/Users/emcummings/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/webrick/httpserver.rb:111:in `service'
/Users/emcummings/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/webrick/httpserver.rb:70:in `run'
/Users/emcummings/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/webrick/server.rb:183:in `block in start_thread'

What am I doing wrong here? If the JSON is too big to parse (I find that sort of hard to believe!) what can I do?

EDIT:

Looks like dropping back to the standard NET::HTTP lib solved my problem:

uri = URI.parse("http://www.reddit.com/r/pics/comments/rhzgc/.json")    
response = Net::HTTP.get_response(uri)
render :json => response.body, :callback => params[:callback]
#OK!

... but I still am curious why I am failing this same call with HTTParty!

Thanks for the help!

score 3 · Answer 1 · answered Jul 23 '12 at 16:39

3

Switched to the OJ JSON parser by adding the following to my Gemfile:

gem 'oj'

and this issue resolved itself.

answered Jul 23 '12 at 16:39

MatthewFord

2,918
2
21
32

This worked great for me too; this should be the accepted answer. – duma Feb 13 '13 at 03:45

score 3 · Answer 2 · answered Feb 13 '13 at 23:08

3

It looks like this can be worked around by calling to_json with the max_nesting option set.

json = obj.to_json(max_nesting: 50)

answered Feb 13 '13 at 23:08

Matt Huggins

81,398
36
149
218

score 3 · Accepted Answer · answered Mar 29 '12 at 03:15

3

It's not that the JSON is too large. It's nested too deeply. HTTParty tries to decode the results that it gets automatically. Following your stack trace, and the HTTParty dependencies, it relies on multi_json which is using the json gem.

Inside of json there is lib/json/pure/parser.rb. The default max depth set is set in there, specifically on line 79. Something in your returned JSON is 20+ levels deep, triggering the exception.

 if !opts.key?(:max_nesting) # defaults to 19
   @max_nesting = 19

answered Mar 29 '12 at 03:15

jdl

17,702
4
51
54

Interesting - I figured it had to do with a json parser somewhere inside of HTTParty - is there a reason for an arbitrary limit? – Evan Mar 29 '12 at 15:26
I'm pretty sure that JSON gem is the same one that nearly everything else uses by default, Rails included. HTTParty is just indirectly requiring it via multi_json. As for why they chose 19 for the default limit, I have no idea. It's an option that can be set, but I'm not sure if HTTParty exposes that easily or not. – jdl Mar 29 '12 at 15:40
HTTParty doesn't seem to expose setting any options on the underlying json parser - [it doesn't pass an options hash(https://github.com/jnunemaker/httparty/blob/master/lib/httparty/parser.rb#L114) into the load or decode methods. – jabley Jul 24 '12 at 09:27

'Nesting too deep' error while retrieving JSON using HTTParty

3 Answers3

Linked