1

I am trying to connect to the Meetup streaming HTTP API and parse the received events in different records. I am using ruby over Sinatra. I chose the 'em-http-request' gem to handle the connection and 'thin' as server. Searching for info on how to handle an API streaming all that I found was about 3 to 6 years old and this is the best example that I found.

In this example the author uses a regex to find the end of each tweet and split them into different records. In my case I didn't find a way to split the stream of meetUp events.

Here is my code:

get '/' do
STREAMING_URL = 'http://stream.meetup.com/2/open_events'
http = EM::HttpRequest.new(STREAMING_URL).get
buffer = ""
http.stream do |chunk|
    buffer += chunk
    while event = buffer.slice!(/{\"utc_offset\"+.../)
        eventRecord = event
        puts eventRecord
    end
end

I open the conection calling to //stream.meetup.com/2/open_events and I start to receive an stream of strings randomly cutted with this format:

{"utc_offset":-14400000,"venue":{"country":"us","city":"Novi","address_1":"43155 Main St Suite 2300N","name":"Game of Clues Escape Room","lon":-83.470833,"state":"MI","lat":42.478107},"rsvp_limit":0,"venue_visibility":"public","visibility":"public","maybe_rsvp_count":0,"description":"<p>Search the Emerald City for clues to help you solve riddles and puzzles to escape the room before the 60 minute timer is up. Work together to complete missions that will bring your group closer together in order to get a clue card.<\/p> \n<p>25.00 per person. Book online at www.gameofclues.com<\/p>","mtime":1467030326494,"event_url":"http:\/\/www.meetup.com\/Escape-Room-Lovers\/events\/232071150\/","yes_rsvp_count":1,"duration":3600000,"payment_required":"0","name":"Game of Clues Escape Room Novi,Mi","id":"232071150","time":1467507600000,"group":{"join_mode":"open","country":"us","city":"Novi","name":"Escape Room Lovers","group_lon":-83.52,"id":20101745,"state":"MI","urlname":"Escape-Room-Lovers","category":{"name":"games","id":11,"shortname":"games"},"group_lat":42.47},"status":"upcoming"}{"utc_offset":14400000,"venue":{"country":"ae","city":"Dubai","address_1":"Jumeirah Lake Towers, outside Dubai Marina Metro","name":"Illuminations Well-Being Center, 409, Fortune Executive Towers, Cluster T, Plot T1, ","lon":55.311668,"lat":25.264444},"rsvp_limit":0,"venue_visibility":"public","visibility":"public","maybe_rsvp_count":0,"description":"<p>Facilitator: Dr. Beryl Bazley<\/p> \n<p>Investment: Free!<\/p> \n<p>For more information call

I tryed to use .slice! over the content of buffer using as param "{"utc_offset"" which is the substring that appears at the beggining of each event but I couldn't figure out how to write the regex that gets everything enclosed between each apeareance of the substring to get as result the whole event.

Also I am not sure that adding the chunks to the variable buffer and then use of the method .slice! it's the best way of getting each event.

Which is the best way of solve this situation?

How can I slice and parse the data received via streaming API?

HERE I ADD THE IMPLEMENTATION OF THE SOLUTION SUGGESTED BY @jordan IN THE COMMENTS:

require 'yajl'
require 'uri' 
require 'yajl/http_stream'


@parser = Yajl::Parser.new(:symbolize_keys => true)

STREAMING_URL = 'http://stream.meetup.com/2/open_events'
Yajl::HttpStream.get(STREAMING_URL, :symbolize_keys => true) do |hash|
    puts hash.inspect
    hash.each {|key, value| puts "#{key} is #{value}" }
end
Ibrah
  • 83
  • 9
  • According to the [API docs](http://www.meetup.com/meetup_api/docs/stream/2/open_events/), “A response message is one HTTP chunk, the body of which is a single json object, described below, terminated by a newline.” When you inspect each chunk, does that appear to the the case? If so, you should just do `JSON.parse(chunk)`. – Jordan Running Jun 27 '16 at 13:49
  • This is `JSON`, right? – Jan Jun 27 '16 at 13:49
  • Barring that, the [em-http-request docs](https://github.com/igrigorik/em-http-request/wiki/Streaming) have this tidbit: “Some API’s, such as Twitter Streaming API provide a JSON stream over a long-lived HTTP connection – a perfect use case for a stream callback. Add a Yajl (JSON) stream parser, and you’re off to the races!” Have you tried that? – Jordan Running Jun 27 '16 at 13:50
  • In any event, it's an understatement to say that trying to parse JSON with regular expressions should be your very last resort. – Jordan Running Jun 27 '16 at 13:55
  • I tried with JSON.parse(chunk) and it works if the chunk content it's a whole event (a complete JSON object), or various events. But in a lot of cases the received chunck is just a part of the event, then I have this error .rvm/rubies/ruby-2.3.0/lib/ruby/2.3.0/json/common.rb:156:in `parse' Because it is trying to take the string and parse it into a hash, but there is not all the info. I didn't know about Yajl(JSON), I will check it. Thanks – Ibrah Jun 27 '16 at 14:26
  • I have used this [gem](https://github.com/brianmario/yajl-ruby) Yajl/JSON and It worked perfectly. I will edit my question to add the working code. Thank You very much. – Ibrah Jun 27 '16 at 15:02

0 Answers0