How do I deploy monorepo code to AWS Lambda using lerna?

Question

I am attempting to make two AWS Lambda functions (written in typescript). Both of these functions share the same code for interacting with an API. In order to not have to copy the same code to two different Lambdas, I would like to move my shared code to a local module, and have both my Lambdas depend on said module.

My initial attempt at staring code between the two lambdas was to use a monorepo and lerna. My current project structure looks like this:

- lerna.json
- package.json
- packages
  - api
    - package.json
  - lambdas
    - funcA
      - package.json
    - func B
      - package.json

lerna.json:

{
  "packages": [
    "packages/api",
    "packages/lambdas/*"
  ],
  "version": "1.0.0"
}

In each of my package.json for my Lambda functions, I am able to include my local api module as such:

"dependencies": {
    "@local/api": "*"
}

With this, I've been able to move the common code to its own module. However, I'm now not sure how to bundle my functions to deploy to AWS Lambda. Is there a way for lerna to be able to create a bundle that can be deployed?

Answer 1

As cp -rL doesn't work on the mac I had to come up with something similar.

Here is a workflow that works if all of your packages belong to one scope (@org):

In the package.json of your lerna repo:

"scripts": {
    "deploy": "lerna exec \"rm -rf node_modules\" && lerna bootstrap -- --production && lerna run deploy && lerna bootstrap"
}

In the package that contains your lambda function:

"scripts":{
    "deploy": "npm pack && tar zxvf packagename-version.tgz && rm -rf node_modules/@org && cp -r node_modules/* package/node_modules && cd package && npm dedupe"
}

Now replace "packagename-version" and "@org" with the respective values of your project. Also add all of the dependent packages to "bundledDependencies".

After running npm run deploy in the root of your lerna mono repo you end up with a folder "package" in the package that contains your lambda function. It has all the dependencies needed to run your function. Take it from there.

I had hoped that using npm pack would allow me to utilize .npmignore files but it seems that that doesn't work. If anyone has an idea how to make it work let me know.

Answer 2

I have struggled with this same problem for a while now, and I was finally forced to do something about it.

I was using a little package named slice-node-modules, as found here in this similar question, which was good enough for my purposes for a while. As I have consolidated more of my projects into monorepos and begun using shared dependencies which reside as siblings rather than being externally published, I ran into shortcomings with that approach.

I've created a new tool called lerna-to-lambda which was specifically tailored to my use case. I published it publicly with minimal documentation, hopefully enough to help others in similar situations. The gist of it is that you run l2l in your bundling step, after you've installed all of your dependencies, and it copies what is needed into an output directory which is then ready to deploy to Lambda using SAM or whatever.

For example, from the README, something like this might be in your Lambda function's package.json:

"scripts": {
  ...
  "clean": "rimraf build lambda",
  "compile": "tsc -p tsconfig.build.json",
  "package": "l2l -i build -o lambda",
  "build": "yarn run clean && yarn run compile && yarn run package"
},

In this case, the compile step is compiling TypeScript files from a source directory into JavaScript files in the build directory. Then the package step bundles up all the code from build along with all of the Lambda's dependencies (except aws-sdk) into the directory lambda, which is what you'd deploy to AWS. If someone were using plain JavaScript rather than TypeScript, they could just copy the necessary .js files into the build directory before packaging.

I realize this question is over 2 years old, and you've probably figured out your own solutions and/or workarounds since then. But since it is still relevant to me, I assume it's still relevant to someone out there, so I am sharing.

Answer 3

Running lerna bootstrap will create a node_modules folder in each "package". This will include all of your lerna managed dependencies as well as external dependencies for that particular package.

From then on, your deployment of each lambda will be agnostic of the fact that you're using lerna. The deployment package will need to include the code for that specific lambda and the node_modules folder for that lambda - you can zip these and upload them manually, or use something like SAM or CloudFormation.

Edit: as you rightly point out you'll end up with symlinks in your node_modules folder which make things awkward to package up. To get around this, you could run something like this prior to packaging for deployment:

cp -rL lambdas/funcA/node_modules lambdas/funcA/packaged/node_modules

The -L will force the symlinked directories to be copied into the folder, which you can then zip.

Answer 4

I have used a custom script to copy the dependencies during the install process.. This will allow me to develop and deploy the application with the same code.

Project structure In the package.json file of the lambda_a, I have the following line:

  "scripts": {
    "install": "node ./install_libs.js @libs/library_a"
  },

@libs/library_a can be used by the lambda code using the following statement:

const library_a = require('@libs/library_a')

for SAM builds, I use the following command from the lmbdas frolder:

export SAM_BUILD=true && sam build

install_libs.js

console.log("Starting module installation")
var fs = require('fs');
var path = require('path');
var {exec} = require("child_process");

if (!fs.existsSync("node_modules")) {
    fs.mkdirSync("node_modules");
}
if (!fs.existsSync("node_modules/@libs")) {
    fs.mkdirSync("node_modules/@libs");
}

const sam_build = process.env.SAM_BUILD || false
libs_path = "../../"
if (sam_build) {
    libs_path = "../../" + libs_path
}


process.argv.forEach(async function (val, index, array) {
    if (index > 1) {
        var currentLib = libs_path + val
        console.log(`Building lib ${currentLib}`)
        await exec(`cd ${currentLib} && npm install` , function (error, stdout, stderr){
            if (error) {
                console.log(`error: ${error.message}`);
                return;
            }
            console.log(`stdout: ${stdout}`);
            console.log('Importing module : ' + currentLib);
            copyFolderRecursiveSync(currentLib, "node_modules/@libs")
        });
    }
});


function copyFolderRecursiveSync(source, target) {
    var files = [];
    // Check if folder needs to be created or integrated
    var targetFolder = path.join(target, path.basename(source));
    if (!fs.existsSync(targetFolder)) {
        fs.mkdirSync(targetFolder);
    }

    // Copy
    if (fs.lstatSync(source).isDirectory()) {
        files = fs.readdirSync(source);
        files.forEach(function (file) {
            var curSource = path.join(source, file);
            if (fs.lstatSync(curSource).isDirectory()) {
                copyFolderRecursiveSync(curSource, targetFolder);
            } else {
                copyFileSync(curSource, targetFolder);
            }
        });
    }
}

function copyFileSync(source, target) {
    var targetFile = target;
    // If target is a directory, a new file with the same name will be created
    if (fs.existsSync(target)) {
        if (fs.lstatSync(target).isDirectory()) {
            targetFile = path.join(target, path.basename(source));
        }
    }
    fs.writeFileSync(targetFile, fs.readFileSync(source));
}