AWS Lambda: How to use tools that must be installed first in linux?

Question

I understand that AWS Lambda runs on the application layer of an isolated environment.

In many situations, functions need to use third-party tools that must be installed first on the linux machine. For example, a media processing function uses exiftool to extract metadata from image, so I install exiftool first.

Now I want to migrate the media processing code into AWS Lambda. My question is, how can I use those tools that I originally must install on linux? My code is written in Java, and exiftool is necessary.

score 3 · Answer 1 · edited Dec 15 '19 at 17:29

To expand on Daniel's answer, if you wanted to bundle exiftool, you would follow steps 1 and 2 for Unix/Linux platforms from the official install instructions. You would then include exiftool and lib in your function's zip file. To run exiftool you would do something like:

const exec = require('child_process').exec;

exports.handler = (event, context, callback) => {
  // './exiftool' gave me permission denied errors
  exec('perl exiftool -ver', (error, stdout, stderr) => {
    if (error) {
      callback(`error: ${error}`);
      return;
    }
    callback(null, `stderr: ${stderr} \n stdout: ${stdout}`);
  });
}

score 1 · Accepted Answer · answered Jul 13 '15 at 18:07

Everything your Lambda function executes must be included in the deployment package you upload.

That means if you want to run Java code, you can reference other Java libraries. (Likewise, if you want to run Node.js code, you can reference other Node libraries.)

Regardless of the tools you use, the resulting .zip file must have the following structure:

All compiled class files and resource files at the root level.

All required jars to run the code in the /lib directory.

(source)

Or you can upload a .jar file.

exiftool, on the other hand, is a Perl command-line program. I suspect that on your local machine, you shell out from your Java code and run it.

You cannot do that in AWS Lambda. You need to find a Java package that extracts EXIF information (I am sure there are plenty to choose from) and include that in your deployment package. You cannot install software packages on Lambda.

Thanks for clarifying this. Now I understand Lambda better. – Jinsong Li Jul 13 '15 at 19:05 — Jinsong Li, Jul 13 '15 at 19:05

score 1 · Answer 3 · edited Jul 07 '16 at 13:58

https://aws.amazon.com/lambda/faqs/

Q: What languages does AWS Lambda support?

AWS Lambda supports code written in Node.js (JavaScript), Python, and Java (Java 8 compatible). Your code can include existing libraries, even native ones. Please read our documentation on using Node.js, Python and Java.

So basically you can call out to native processes if they are pre-installed but only from JavaScript and Java as the parent process.

To get a rough idea of what is installed have a look at what packages are installed:

https://gist.github.com/royingantaginting/4499668

This list won't be a 100% accurate, to do that you would need to look directly at the AMI image (ami-e7527ed7)

exiftool doesn't appear to be installed by default. I doubt the account running the lambda function would have enough rights to install anything globally but you could always bundle exiftool with your Node or Java function.

You may also want to have a look at lambdash (https://github.com/alestic/lambdash) which allows you to run command from your local command line on a remote lamdba instance

score 0 · Answer 4 · answered Jan 17 '20 at 14:05

0

This can now be done using AWS Lambda Layers.

An example of how to prepare a layer for exiftool specifically can be found here:

https://gist.github.com/hughevans/6b8c57839b8194ba910428de4375794a

answered Jan 17 '20 at 14:05

patforna

721
6
8

AWS Lambda: How to use tools that must be installed first in linux?

4 Answers4