MessagePack Perl to C++ deserialisation

Question

I'm new to messagepack, and I'm trying to take a hash in perl, serialize it using messagepack, write it to a file, pass that to c++ code where the file is read and deserialized into a map.

My perl code to generate the file is (note - I added an extra part to check I could read the file back in and deserialize it correctly in perl, although I don't really need to do this):

#! perl

use strict;
use warnings;

use Data::MessagePack;

my %hTestHash = ('AAAAAA' => '20020101',
                 'BBBBBB' => '20030907');

my $packed = Data::MessagePack->pack(\%hTestHash);

open my $fh, '>', 'splodge.bin' or die "Failed to open slodge.bin for write: $!";
print $fh $packed;
close $fh;

open my $fh2, '<', 'splodge.bin' or die "Failed to open slodge.bin for read: $!";
local $/;
my $file = <$fh2>;

my $hrTest = Data::MessagePack->unpack($file);

My c++ code to deserialize is then:

#include "msgpack.hpp"
#include <string>
#include <iostream>
#include <sstream>
#include <fstream>

int main(void)
{
  // Deserialize the serialized data.
  std::ifstream ifs("splodge.bin", std::ifstream::in);
  std::stringstream buffer;
  buffer << ifs.rdbuf();
  msgpack::unpacked upd;
  msgpack::unpack(&upd, buffer.str().data(), buffer.str().size());
  msgpack::object obj = upd.get();
  std::map<std::string, std::string> output_map;
  msgpack::convert(output_map, obj);

  string date = output_map.at("AAAAAA");

  return 0;
}

This produces a 2 element map in output_map, but it only contains garbage values - my program crashes out on output_map.at() with

{"▒▒▒▒▒▒"=>"▒▒▒▒▒▒▒▒", "▒▒▒▒▒▒"=>"▒▒▒▒▒▒▒▒"}
terminate called after throwing an instance of 'std::out_of_range'
  what():  map::at
Aborted

I've been unable to find any examples of this particular use case, and struggling to see what's wrong - is this a problem at the serialization end or (seems more likely) at the deserialization end?

EDIT: Thanks to @SinanÜnür for pointing out my mistake, which I have now updated in the question. This doesn't change the fact that the hash is being populated with garbage values, so the same exception is being thrown regardless of the key being searched for.

@SinanÜnür so this same code is working correctly for you and producing a map in c++? — rbennett485, Dec 02 '16 at 16:49
`'>'` and `'<'` should be `'>:raw'` and `'<:raw'`, especially if you are on a Windows machine. Similarly, the C++ program should set the stream to binary. — ikegami, Dec 02 '16 at 17:30
@SinanÜnür I'm guessing you downvoted my answer and deleted your own, as they happened very close to one another. Would you mind reinstating it? It was a useful part of arriving at the solution — rbennett485, Dec 15 '16 at 17:32
The answer you provided to the question involves information you did not share when you posed the question. Therefore no one but you had any chance of correctly answering the question and it is likely no one will benefit from the question nor the answer. — Sinan Ünür, Dec 15 '16 at 18:45
I'm not sure that's a productive attitude, but then this probably isn't the place for discussing it either. — rbennett485, Dec 15 '16 at 18:51

rbennett485 · Accepted Answer · 2016-12-15T11:53:27.903

Finally managed to get it working with a combination of proper binary file reading and some munging with weird in-house data types that we have.

The functioning code (which deserializes the structure into a Map<string, msgpack::object>, requiring an extra step of deserializing each value when it is needed) is:

#include "msgpack.hpp"
#include "map.h"
#include <string>
#include <iostream>
#include <fstream>

void unpack_map(const msgpack::object& o, Map<string,msgpack::object>& results){
  ASSERT(o.type == msgpack::type::MAP);
  for (msgpack::object_kv *p = o.via.map.ptr, *pend = (o.via.map.ptr + o.via.map.size); p != pend; ++p)
    results[p->key.as<string>()] = p->val;
}

int main(void)
{
  streampos size;
  char * memblock;

  ifstream file ("splodge.bin", ios::in|ios::binary|ios::ate);
  if (file.is_open())
    {
      size = file.tellg();
      memblock = new char [size];
      file.seekg (0, ios::beg);
      file.read (memblock, size);
      file.close();

      cout << "the entire file content is in memory";
    }
  else cout << "Unable to open file";

  msgpack::unpacked upd;
  msgpack::unpack(&upd, memblock, size);
  msgpack::object obj = upd.get();

  if(obj.type != msgpack::type::MAP) {
    cout << ":(" << endl;
  } else {
    cout << ":)" << endl;
  }
  cout << "size: " << obj.via.map.size << endl;

  Map<string, msgpack::object> objectMap;
  unpack_map(obj, objectMap);

  msgpack::object val  = objectMap["BBBBBB"];
  string tmp = string(val.via.raw.ptr, val.via.raw.size);

  delete[] memblock;
  return 0;
}

where tmp takes the value 20030907

MessagePack Perl to C++ deserialisation

1 Answers1