4

I'm trying to use the Sharp library in AWS Lambda but it requires the module be compiled for the lambda environment. The instructions say to create an ec2 instance and compile it on there - but I noticed that there are a few tools to help with that but they are all at least a year old with no maintenance. Is there a package that comes with Serverless, or something that's considered the standard way now?

I've found these but they are all at least a year old since a commit

https://github.com/node-hocus-pocus/thaumaturgy

https://github.com/Max-Kolodezniy/aws-lambda-build

https://github.com/tomdale/lambda-packager

Maybe there is a directory somewhere where I can just download a precompiled Sharp library for AWS lambda?

Nik Sumeiko
  • 8,263
  • 8
  • 50
  • 53
MonkeyBonkey
  • 46,433
  • 78
  • 254
  • 460
  • 1
    I think you will need to compile it in Amazon Linux ec2 instance unless someone else has already done it and published for others – Ashan Mar 27 '17 at 02:34
  • This repository https://github.com/adieuadieu/serverless-sharp-image/tree/master/lib claims it has a tarball with `node_modules/`, that `sharp` needs to run on Nodejs 4.3.2 (as on Lambda environment). – Nik Sumeiko Mar 27 '17 at 08:10
  • @NikSumeiko yeah that looks like a good resource to download from, thanks. A good open source project might be to create a repository with a directory of all the lambda specific builds for libraries that need it. – MonkeyBonkey Mar 27 '17 at 11:38

4 Answers4

3

I made it work using sharp-0.17.3-aws-linux-x64-node-6.10.1.tar.gz tarball, that was created on AWS EC2 instance running Nodejs 6.10.1. The tarball contains node_modules/ directory with sharp system binaries (libvips library) specific to the Lambda execution environment.

 

Project structure

To avoid conflicts between my local node_modules/ (Nodejs 7.5 on Mac) and node_modules/ inside the tarball (Nodejs 6.10 on Linux), I'm creating my Lambda service under a subdirectory.
Project structure looks as follows:

node_modules/
service/
  node_modules/ <= sharp-0.17.3-aws-linux-x64-node-6.10.1.tar.gz
  utils/
  handler.js
  package.json  <= engines: node 6.10.1
  serverless.yml
src/
jasmine.json
package.json

Most of the dependencies I need are for development and testing purpose. These are maintained inside root package.json file (also includes sharp, but compiled for my Nodejs 7.5 environment, offering to test image manipulations locally).

My service/handler.js and service/utils/ contains ES6 compatible source code with Lambda function handler – it is transpiled from src/ directory.

If I need other dependencies for production (besides sharp), I install them to services/package.json using --prefix option. But not aws-lambda, neither aws-sdk – they are globally installed within Lambda, meaning no need to include them in deployable .zip file.

npm i -S lodash --prefix services/

It ensures installation of lodash version compatible with Lambda environment, because service/package.json defines Nodejs version to rely on:

{
  "private": true,
  "engines": { "node" : "6.10.1" },
  "dependencies": {
    ...
  }
}

However, there's a nuance — other production dependencies doesn't have to be environment dependent. If so, they won't work, because you install them from your local machine, which isn't equal to Lambda's one.

 

Lambda function deployment

Since Lambda requires .zip archive, I compress contents of my service/ directory. And my Lambda functions works. Everything is ES6 compatible, sharp has Lambda environment binaries and my other production dependency versions correlates with Nodejs 6.10.1.

 

Additionally, I'd suggest to use Serverless ⚡️ (I use it too). It dramatically simplifies Lambda functions development and deployment.

Nik Sumeiko
  • 8,263
  • 8
  • 50
  • 53
  • Thanks for detailed description Nik! I am someone that is relatively new to lambda, serverless framework, etc. If you have a moment, can you please explain step by step how to 'use the tarball' file? I assume you are referring to this [tarball](https://github.com/adieuadieu/serverless-sharp-image/blob/master/lib/sharp-0.17.3-aws-linux-x64-node-6.10.1.tar.gz). Do you simply double click on that file within your /services folder and the node_modules is then created? I am attempting this approach, then deploying it via serverless framework but no luck so far. – andre Apr 12 '17 at 21:19
  • The code within my handler works well locally. It's just figuring out how to get the dependencies right so it works on a deployed lambda function. – andre Apr 12 '17 at 21:23
  • I got it to work! :-) The sharp API changed a bit from 0.16 to 0.17 so after making some adjustments in my logic, everything works well – andre Apr 13 '17 at 02:00
  • @AndriyKulak good to hear it worked for you at the end. Based on your comments, I understood it's not obvious what tarball I refer to under "Project structure" section. Therefore, I've added a link to a "tarball" reference now – it's the same as at a very top of my answer. – Nik Sumeiko Apr 13 '17 at 09:08
  • eghad - is there an easier way? Why can't they fix their library so it doesn't require jumping through such hoops? – Scott Oct 03 '19 at 15:31
1

Nik's answer definitely helped me to get to a working solution! One thing that I would add is that the people behind serverless-sharp-image updated their package so the tarball works with node v6.10 now so I don't see a reason to have two different node environments referenced. I do everything in v6.10.

https://github.com/adieuadieu/serverless-sharp-image/tree/master/lib

andre
  • 1,660
  • 3
  • 19
  • 31
  • Are you sure your local environment is equal to [Lambda execution environment](http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html)? The thing is, it's not only Nodejs version equality that matter. If I'm not wrong, `libvips` library (a part of `sharp`) differentiate between Linux, Windows, Mac machines. Lambda run on Linux. Therefore, `node_modules/sharp` shall contain `libvips` library installed on Linux. – Nik Sumeiko Apr 13 '17 at 09:15
  • yes. my folder structure is very similar to yours @NikSumeiko. I just use node v6.10 in /service folder since the tarball supports that. The only thing that slightly bothers me is that I cannot easily iterate in my local environment. I haven't been able to get [serverless-offline](https://github.com/dherault/serverless-offline) npm package to work with that directory structure. – andre Apr 13 '17 at 14:37
1

Had similar problem and managed to install binaries for Linux x64 platform by

npm install --arch=x64 --platform=linux --target=8.10.0 sharp

Then just upload Lambda as usual and it works just fine. Above works on Mac as well as windows and details are in documentation at http://sharp.pixelplumbing.com/en/stable/install/#aws-lambda

spirytus
  • 10,726
  • 14
  • 61
  • 75
1

For anyone stumbling upon this post now. I've accomplished this by copying my package.json file into an AWS Cloud9 IDE and simply running npm install. From there, just download the node_modules/ folder.

Dillon Bourne
  • 41
  • 5