Adding zip file as Content in Web API response doubling file size on download

Question

I am saving zip files to an AWS S3 bucket. I am now trying to create a C# .NET API that will allow me to download a specified key from the bucket and save it to a HttpResponseMessage in the Content key.

I've referred to the following question to set up my response for zip files: How to send a zip file from Web API 2 HttpGet

I have modified the code in the previous question so that it instead reads from a TransferUtility stream.

Problem is I am coming into an error when trying to extract or view the file that looks like the following:

The response I am getting back from the API looks like:

The relevant code looks like:

[HttpGet, Route("GetFileFromS3Bucket")]
public HttpResponseMessage GetFileFromS3Bucket(string keyName)
{
    HttpResponseMessage response = new HttpResponseMessage();
    string bucketName = "myBucket";
    RegionEndpoint bucketRegion = RegionEndpoint.ARegion;
    IAmazonS3 s3Client;
    s3Client = new AmazonS3Client(bucketRegion);

    try
    {
        var fileTransferUtility = new TransferUtility(s3Client);
        var stream = fileTransferUtility.OpenStream(bucketName, keyName);
        response.Content = new StreamContent(stream);
        response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
        response.Content.Headers.ContentDisposition.FileName = keyName + ".zip";
        response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/zip");
        response.StatusCode = HttpStatusCode.OK;
    }
    catch (Exception e)
    {
        response.Content = new StringContent("Something went wrong, error: " + e.Message);
        response.StatusCode = HttpStatusCode.InternalServerError;
    }

    return response;
}

Results of troubleshooting:

• The file from the Web API comes out with nearly double the expected size based on what is in S3. This is consistent across different files
• Changing the bucket to be publicly accessible did not help (setting since reverted to not allowing public access)
• Changing the file type to XML did not display a nicely formatted error (there was a suggestion that you may receive an XML response if an error was provided from S3)
• Saving the S3 stream to a file and then saving directly to a file resulted in the correct file size. Seems safe to say the stream from S3 is not the problem

It appears that there ia a problem with the way the HTTPResponseMessage is handling the zip file. I'm unsure of whether it is actually on the server side, or whether it is on the client to parse the data and Swagger is simply incapable of doing that. Any help would be greatly appreciated.

Update 1 I do not believe this string is Base64 encoded as the result I got from converting the stream to a string is the following:

I've updated the code sample with the two lines showing the conversion from a stream to string.

Update 2 I've confirmed the issue is with how the response is handling the stream, or something in the response itself. Downloading the file stream from S3 and saving to a new file on the local computer resulted in a valid file that opened as expected.

Update 3 Link to GDrive folder with testing files: https://drive.google.com/drive/folders/1q_N3NTHz5E_nebtBQJHor3HfqUZWhGgd?usp=sharing I unfortunately can't provide access to the original file as it contains sensitive data. The provided files are still causing the same problem however. Interesting to note that the test file came out looking like:

The underscores on either side of the filename are quite strange.

I am running the following relevant packages:

Update 4 I've found the following UTF8 references in various files:

File: configuration91.svcinfo

I could not find anything that said anything about 'responseEncoding' anywhere in the project.

Answer 1

I am going to throw an answer up, because what's happening to you is unorthodox. I use S3 for many things and have done what you are doing with no problems in the past. To ensure that I am mimicking what you are doing, I duplicated your code:

[HttpGet, Route("GetFileFromS3Bucket/{keyName}")]
public HttpResponseMessage GetFileFromS3Bucket(string keyName)
{
    string bucketName = "testzipfilesagain";
    string awsAccessKey = "AKIAJ********A3QHOUA";
    string awsSecretKey = "IYUJ9Gy2wFCQ************dCq5suFS";

    IAmazonS3 client = new AmazonS3Client(awsAccessKey, awsSecretKey, RegionEndpoint.USEast1);

    var fileTransferUtility = new TransferUtility(client);
    var stream = fileTransferUtility.OpenStream(bucketName, "md5.zip");

    var resp = new HttpResponseMessage();

    resp.Content = new StreamContent(stream);
    resp.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
    resp.Content.Headers.ContentDisposition.FileName = keyName + ".zip";
    resp.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/zip");
    resp.StatusCode = HttpStatusCode.OK;

    return resp;
}

These are the packages I have installed:

  <ItemGroup>
    <PackageReference Include="AWSSDK.S3" Version="3.3.111.37" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.WebApiCompatShim" Version="2.2.0" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="5.5.1" />
  </ItemGroup>

Everything runs perfectly well.

Trying to troubleshoot your code is going to be fruitless because it works perfectly fine, but there is something wrong with your environment.

So this isn't an answer to your question, but a answer to how you can try to solve the issue at hand and get past this.

1. Make sure your nuget packages are up to date
2. Do you have any middleware injected in your pipeline? If so, what?
3. Post your startup.cs -- maybe something is out of order in your Configure routine.
4. Could you start a brand new project and try your code in that?
5. Can you try a small 5KB zip file and post the original and the corrupt so we can look?

I would love to get to the bottom of this as I really like to solve these types of problems.

---

EDIT 1

So I looked at the zip files and they have been run through a UTF8 encoding process. So, if you take your original zip file, and run this code on it:

    var goodBytes = File.ReadAllBytes("Some test to upload to S3.zip");
    var badBytes = File.ReadAllBytes("_Some test to upload to S3.zip.zip_");

    File.WriteAllText("Some test to upload to S3.zip.utf8", Encoding.UTF8.GetString(goodBytes));
    var utf8EncodedGoodBytes = File.ReadAllBytes("Some test to upload to S3.zip.utf8");

    var identical = badBytes.SequenceEqual(utf8EncodedGoodBytes);

It the results are:

I am going to do some research and figure out what could be causing your stream to become UTF-8 encoded. Is there anything in your config that looks like this? Can you search your entire solution for anything that resembles "utf" or "utf8" or "utf-8"?