How can I read this binary XML response with Ruby?

Question

I'm trying to read a response from a marketplace web service. Every other response from this web service is returned in XML format. However, this particular call is requesting a file download. I'm unfamiliar with the way in which it's returned. After looking at the contents, there is XML as well as encoded binary data which is in there as some sort of attachment.

The request I make looks like this. The request is a simple XML request:

  begin
    response = Net::HTTP.start(url.host, url.port, :use_ssl => url.scheme == 'https') do |http|
      http.verify_mode = OpenSSL::SSL::VERIFY_NONE
      http.request(request)
    end
  rescue Errno::ECONNRESET => e
    count += 1
    retry unless count > 10
    puts "Tried 10 times and couldn't get #{url.host}: #{e}"
  end

Here is what the response.body looks like:

--MIMEBoundaryurn_uuid_AF2837F4196B2631EC15070889135182607126
Content-Type: application/xop+xml; charset=utf-8; type="text/xml"
Content-Transfer-Encoding: binary
Content-ID: <0.urn:uuid:AF2837F4196B2631EC15070889135182607127>

<?xml version='1.0' encoding='UTF-8'?>
<downloadFileResponse xmlns="http://www.marketplace.com/marketplace/services">
  <ack>Success</ack>
  <version>1.1.0</version>
  <timestamp>2017-10-04T03:48:33.518Z</timestamp>
  <fileAttachment>
    <Size>25895</Size>
    <Data><xop:Include xmlns:xop="http://www.w3.org/2004/08/xop/include" href="cid:urn:uuid:E3A8215C82DBC51E6D1507090865513"/></Data>
  </fileAttachment>
</downloadFileResponse>

--MIMEBoundaryurn_uuid_AF2837F4196B2631EC15070889135182607126
Content-Type: application/zip
Content-Transfer-Encoding: binary
Content-ID: <urn:uuid:E3A8215C82DBC51E6D1507090865513>

PK&úCKÃEK¬gdi∂6509805153_report.xmlUT   hH‘YhH‘Yux00ÏùÎs‚8∂¿ø˜_°€üÓ≠=-â Ù6ÚÍ< ›”≥µïr@   NåÕ⁄&è˘ÎØÑÅêàçÕ‡…¶+ùnåd=|Œ—œÁË±ÛœáæÓT:æ˜Îg¥?Âu¸Æ„]ˇ˙y]ïƒÁ~˘¥≥;tokvd◊:=€ªVM|/T!–˘ΩP'
º≤∫¥Àˆ¿ Àj˜x◊U’ÔÎT ã¬œ˙-ﬂÌ6’¿¢/ü>Ìú]‘Td;n¯e¸Ò∞ˆ L%‰åqÀ*!é, òdB±≥=Ic™Û®ÇÛpÙ©ÁªÆ≥ŸA∏≥=˚≈8ŸûÑ—©›W_v˛Á_’Z•]˘W€$Ó‹˛˚ﬂ_∆9û“å3€/Ûòbº)œ4…8KΩØõÚî>äÀË≈Ÿ˛Ít\ÿ›Í¯˝ß{ƒy∆7hÙt_=›ÄC$·Ä3åürƒâtgˆúASuúÅ£ªwnÎ¿lç_'èo—ä•"ÙîAÇ)–ÆÔ{˜ v£ÿuÔ∫”õL2Ãf«œæ√„Ô™NÙ¯ºb{∂\Ÿ”{MSLnfGÍ,h˛ù„ufÚ}ØÃˇ<Mú≥·áëÌV˝ÓL&åuKJÿBhöy&Ÿ∏ä˛–ãÇµ<o=UpÊ˚8«@ÎEKwŒl˝Œ-∞Ë¥O›4õÓ”N√~ÏÎ~Ø∫ T∑ÌË€aàx   ¡$mƒ 
...
--MIMEBoundaryurn_uuid_AF2837F4196B2631EC15070889135182607126--

Obviously I can see there is zip data here. But I've read every other response with Hash.from_xml and obviously that's not going to work here.

Update If I write the string to a file test.zip, I can unzip it at the Linux command line, and it creates a readable XML file after flashing this warning:

Archive:  test.zip
warning [test.zip]:  822 extra bytes at beginning or within zipfile
  (attempting to process anyway)
  inflating: 6509805153_report.xml

Not sure what the extra bytes it's complaining about are.

Update 2 That's definitely the MIME header and the XML envelope. I can confirm that if I strip those characters out by hand, and the MIME footer, then the test file unzips without warning.

So this appears to be a zipped XML envelope containing a zip file.

_"I've read every other response with `Hash.from_xml` [...]"_ – could you give a more complete example, please? Where is your response in that call? — Stefan, Oct 04 '17 at 12:54
Hi Stefan. Please see my updates above as I think they shed light on what I've discovered. The other responses I've read don't really matter because they come in as straight XML string data, whereas this one contains those MIME headers. — RubyRedGrapefruit, Oct 04 '17 at 13:16
According to the `Content-Type`, this is [XML-binary Optimized Packaging](https://www.w3.org/TR/xop10/). It allows to extract certain binary parts from the XML to serialize the data more efficiently. The `` has to be replaced by the 2nd MIME part. There's probably a library for that. — Stefan, Oct 04 '17 at 13:35
Couldn't you just unzip the string? https://stackoverflow.com/questions/5871498/ruby-unzip-string — mahemoff, Oct 04 '17 at 13:49
No. I have tried that code, and it fails silently. The ```puts``` statement is not reached. — RubyRedGrapefruit, Oct 04 '17 at 14:17
It appears that there is no library to read this data, unfortunately. I can't even find any resources from the past 6 years in a Google search. This is for the eBay File Transfer API. It's really hard for me to believe that some such library doesn't exist. I'll have to come up with a hack whereby I coerce whatever ```unzip``` is installed on the server to do this, which means I'll have to write to a temp file. Ugly. — RubyRedGrapefruit, Oct 04 '17 at 14:19

score 1 · Accepted Answer · answered Oct 04 '17 at 14:24

This may be a bit more complicated than just using a library. Closest I could find was an implementation in the savon SOAP library that handles xop in multipart responses. You might be able to analyze the code there and come up with a solution that fits your need, or if this is a SOAP service, leverage the savon gem.

https://github.com/savonrb/savon-multipart/blob/master/lib/savon/multipart/response.rb#L63-L80

How can I read this binary XML response with Ruby?

1 Answers1