Serializing ProtoBuf through IHttpHandler yields invalid results

Question

I am using ASP.NET WebForms, which means I will be sending Protobuf data via ASPX or ASHX pages.

I am attempting to build a GTFSRT file, for which there is a Nuget package called GTFSRealTimeBindings. This uses Protobuf and Protobuf-net to compress and send the data.

The issue that I am having is that when the data is sent, something is getting jumbled, so it can't be read on the receiving end, and I am not sure how to correct it. I think it is in the encoding, but I am not setting that, so I am not sure how to change it.

What I ended up doing is writting an http handler (ashx page) that will download a GTFS file from another source, and then simply try to forward it. I know that the GTFS file can be read and de-coded from the other source. But every time I try to server the file from my ashx page, I can't decode the protbuf object.

Here is a very basic code set:

public class Vehicles : IHttpHandler
{

  public void ProcessRequest(HttpContext context)
  {

      WebRequest req = HttpWebRequest.Create("https://cdn.mbta.com/realtime/VehiclePositions.pb");
      FeedMessage feed = Serializer.Deserialize<FeedMessage>(req.GetResponse().GetResponseStream());

      Serializer.Serialize(context.Response.OutputStream, feed);
  }
}

In this snippet, you will note that I am downloading a Protobuf file from cdn.mbta.com, then simply trying to take the result that I got, and pass it back down.

When I try to read this into my sample application:

WebRequest req = HttpWebRequest.Create("http://localhost:54988/Secure/Admin/Reports/GtfsRt/Vehicles.ashx");
FeedMessage feed = Serializer.Deserialize<FeedMessage>(req.GetResponse().GetResponseStream());

The message I get is: 'Invalid wire-type; this usually means you have over-written a file without truncating or setting the length; see http://stackoverflow.com/q/2152978/23354'

If I run Fiddler while this page is hit, I notice that the response I get from cdn.mbta.com is different than the response that this page gives (minus the headers of course).

For example, the first two lines from mbta.com in fiddler show:

2.0 ] y1601"T

But the first two lines from my response are: 2.0 W y1601"N

Any ideas on what is causing this, and how I can correct it? I have tried setting the encoding using

content.Response.ContentEncoding=Encoding.Utf8

I and I went through and tried all of the other encodings to try to set it correctly.

======UPDATE====== In response to Marc's question, I have taken the Base64 string of the response payload, and the response from the first source does not match the response once I forward it on.

Response from mbta.com (limited to first few characters): Cg0KAzIuMBAAGI/e8eIFEl0KBXkwNzIzIlQKHAoIMzkyNTAwNjcqAjg4MAAaCDIwMTkwMjA3IABCDg

Response from my service (limited to first few characters): CgsKAzIuMBiP3vHiBRJXCgV5MDcyMyJOChgKCDM5MjUwMDY3GggyMDE5MDIwNyoCODgSFA3skilCFQ

As you can see, they are different. I will be working on a solution I can upload to demo the issue. Thank you again!

Answer 1

I think I have been able to come up with a solution.

My Web Code turned into the following:

    public void ProcessRequest(HttpContext context)
    {
        context.Response.Clear();
        context.Response.Buffer = true;
        context.Response.AddHeader("content-disposition", "attachment;filename=VehiclePositions.pb");
        context.Response.ContentType = "application/x-protobuf";
        WebRequest req = HttpWebRequest.Create("https://cdn.mbta.com/realtime/VehiclePositions.pb");
        FeedMessage feed = Serializer.Deserialize<FeedMessage>(req.GetResponse().GetResponseStream());
        using (MemoryStream ms = new MemoryStream())
        {
            Serializer.Serialize(ms, feed);
            context.Response.BinaryWrite(ms.ToArray());
            context.Response.End();
        }
    }

Not sure if it was related or not, but as I was debugging I noticed that sometimes the request wouldn't hit the web-service. Removed authentication, and it can now decode the message. I believe this was working before, as I was stepping through the web-code, but it is something to be aware of. Thank you again Marc!

Answer 2

What the response looks like in fiddler becomes tricky if you aren't looking at binary; your best bet here is to look on the HexView tab; everything in green is the headers; everything in black is the payload.

Using your original url of https://cdn.mbta.com/realtime/VehiclePositions.pb, I note that this is ~26k, but with 66591 payload bytes, using gzip internally; so you need to make sure you accept the offer of Fiddler to decode that first - then I did a "copy as base-64" in fiddler (of just the payload, not the headers, in the HexView tab) and ran it through https://protogen.marcgravell.com/decode (using the base-64 option), and it parsed through to the end.

So: what I would suggest is:

• do a similar request to your handler
• check the length - is it 66k? or 26k? or something else? and if it is 26k, is it marked as gzip-encoded?
• when you copy the payload base-64 for each (gzip-decoded, if necessary): is the base-64 the same?
• what does https://protogen.marcgravell.com/decode say about the base-64 from your handler? does it accept it and parse through to the end?

This should help you identify the problem; or help me to identify it with you.