Extract a KMS-encrypted ZIP file from AWS S3

Question

I want to use a Lambda function to retrieve a specific ZIP file from AWS S3, decrypt it and extract it.

Here is the code I have:

const AWS = require('aws-sdk');
const zlib = require('zlib');
const fs = require('fs');
const stream = require('stream');

exports.handler = function (event, context) {
  const jobInfo = event['CodePipeline.job'].data;
  const artifactsInfo = jobInfo.inputArtifacts[0].location;
  const bucket = artifactsInfo.s3Location.bucketName;
  const key = artifactsInfo.s3Location.objectKey;

  const credentials = jobInfo.artifactCredentials;
  const s3 = new AWS.S3({
    credentials: credentials,
  });
  const kms = new AWS.KMS({
    credentials: credentials,
    region: 'eu-central-1',
  });

  s3.getObject({
    Bucket: bucket,
    Key: key,
  }, function(err, data) {
    if (err) {
      // context.done(err);
      console.error(err);
      return;
    }

    console.log('Received file', key);

    const buff = new stream.PassThrough();

    kms.decrypt({CiphertextBlob: data.Body}, function(err, decryptData) {
      if (err) {
        console.error(err);
        return;
      }

      buff.end(decryptData.Plaintext);

      console.log('Decoded S3 object encrypted with KMS ID', decryptData.KeyId);

      buff
      .pipe(zlib.createGunzip())
      .on('error', console.error)
      .on('entry', function(entry) {
        console.log(entry);
      });
    });

  });
};

However, the ZIP file is like 5MiB and I get the following error from the KMS request:

ValidationException: 1 validation error detected: Value 'java.nio.HeapByteBuffer[pos=0 lim=128011 cap=128011]' at 'ciphertextBlob' failed to satisfy constraint: Member must have length less than or equal to 6144
    at Request.extractError (/home/victor/dev/s3-zip-extract/node_modules/aws-sdk/lib/protocol/json.js:48:27)
    at Request.callListeners (/home/victor/dev/s3-zip-extract/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/home/victor/dev/s3-zip-extract/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/home/victor/dev/s3-zip-extract/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/home/victor/dev/s3-zip-extract/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/victor/dev/s3-zip-extract/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /home/victor/dev/s3-zip-extract/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/home/victor/dev/s3-zip-extract/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/home/victor/dev/s3-zip-extract/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/home/victor/dev/s3-zip-extract/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
  message: '1 validation error detected: Value \'java.nio.HeapByteBuffer[pos=0 lim=128011 cap=128011]\' at \'ciphertextBlob\' failed to satisfy constraint: Member must have length less than or equal to 6144'

How could I deal with this? Thank you!

Was your S3 object encrypted client-side when it was uploaded? Or are you using server-side S3 encryption with a KMS key? — Matt Houser, Oct 08 '17 at 17:51
@MattHouser see my edit please! I have updated my situation! — Victor, Oct 09 '17 at 08:35
you have now changed the nature of your question (the title does not match your problem now). Please (a) provide your own answer to your original question that you have now solved, and (b) ask your new problem as a new Stack Overflow question. This way, people finding these questions in the future will find the appropriate answers clearly. — Matt Houser, Oct 10 '17 at 14:31

Victor · Accepted Answer · 2018-12-25T22:04:16.647

After a deeper dive into the documentation, I found out I did not have to decrypt the object myself as it came plaintext to the client. I removed the decryption step and I got to a point where my code looked like this:

buff.end(data.Body);
buff
  .pipe(zlib.createGunzip())
  .on('error', console.error)
  .on('entry', function(entry) {
    console.log(entry);
  });

Note (I am adding this because it took me some time to figure it out). Amazon exports its .zip files to PKZIP format which zlib is unable to work with.

Extract a KMS-encrypted ZIP file from AWS S3

1 Answers1