How do I get relative package.json dependencies to work with AWS's sam build command on Windows?

Question

My goal is to share library code among several of our lambda functions using layers and be able to debug locally and run tests.

npm is able to install dependencies from the local file system. When we change our library code, we want all users of that library to get the updated code without having to set up a dedicated npm server. I'm able to debug locally just fine using the relative paths, but that's before I involve sam build. sam build creates a hidden folder at the root of the repository and builds the folder out and eventually runs npm install, however this time the folder structure is different. The relative paths used in the package.json file are now broken. We can't use explicit paths because our repositories reside under our user folders, which are of course different from one developer to another.

Here's what I did:

I created a project using sam init and took the defaults (except the name of sam-app-2) for a nodejs 12.x project (options 1 and 1).

That command created a folder called sam-app-2 which is the reference for all of the following file names.

I created a dependencies/nodejs folder.

I added dep.js to that folder:

exports.x = 'It works!!';

I also added package.json to the same folder:

{
  "name": "dep",
  "version": "1.0.0",
  "description": "",
  "main": "dep.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

Under hello-world (the folder housing the lambda function), I added the following to the dependencies in package.json:

"dep": "file:../dependencies/nodejs"

I ran npm install under hello-world and it copied the dependencies under node_modules/dep. Normally, you would't do that here. This is purely to allow me to run locally without involving the sam CLI. It's just pure nodejs code. I can run tests, I can debug and not have to wait twenty seconds or more while sam packages up everything and invokes my function. Developing in this state is awesome because it's very fast. However, it'll eventually need to run correctly in the wild.

I edited ./hello-world/app.js:

const dep = require('dep');
let response;

exports.lambdaHandler = async (event, context) => {
    try {
        // const ret = await axios(url);
        response = {
            'statusCode': 200,
            'dep.x': dep.x,
            'body': JSON.stringify({
                message: 'Hello, World!!',
                // location: ret.data.trim()
            })
        }
    } catch (err) {
        console.log(err);
        return err;
    }

    return response
};

if(require.main === module){
    (async () => {
        var result = await exports.lambdaHandler(process.argv[1]);
        console.log(result);
    })();
}

I edited template.yml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app-2

  Sample SAM Template for sam-app-2

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x
      Layers:
        - !Ref DepLayer
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

  DepLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
        LayerName: sam-app-dependencies-2
        Description: Dependencies for sam app [temp-units-conv]
        ContentUri: ./dependencies/
        CompatibleRuntimes:
          - nodejs12.x
        LicenseInfo: 'MIT'
        RetentionPolicy: Retain
Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

Running it straight from the command line works:

sam-app-2> node hello-world\app.js
{
  statusCode: 200,
  'dep.x': 'It works!!',
  body: '{"message":"Hello, World!!"}'
}

Even sam deploy works! Yes, it deploys the code to the cloud and when I invoke the lambda function in the cloud, it gives the same result as above.

However, when I run sam build, it fails with:

Building resource 'HelloWorldFunction'
Running NodejsNpmBuilder:NpmPack
Running NodejsNpmBuilder:CopyNpmrc
Running NodejsNpmBuilder:CopySource
Running NodejsNpmBuilder:NpmInstall

Build Failed
Error: NodejsNpmBuilder:NpmInstall - NPM Failed: npm ERR! code ENOLOCAL
npm ERR! Could not install from "..\dependencies\nodejs" as it does not contain a package.json file.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\Brandon\AppData\Roaming\npm-cache\_logs\2020-03-04T19_34_01_873Z-debug.log

When I try to invoke the lambda locally:

sam local invoke "HelloWorldFunction" -e events/event.json
Invoking app.lambdaHandler (nodejs12.x)
DepLayer is a local Layer in the template
Building image...
Requested to skip pulling images ...

Mounting C:\Users\Brandon\source\repos\sam-app-2\hello-world as /var/task:ro,delegated inside runtime container
2020-03-03T19:34:28.824Z        undefined       ERROR   Uncaught Exception      {"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'dep'\nRequire stack:\n- /var/task/app.js\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js","stack":["Runtime.ImportModuleError: Error: Cannot find module 'dep'","Require stack:","- /var/task/app.js","- /var/runtime/UserFunction.js","- /var/runtime/index.js","    at _loadUserApp (/var/runtime/UserFunction.js:100:13)","    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)","    at Object.<anonymous> (/var/runtime/index.js:43:30)","    at Module._compile (internal/modules/cjs/loader.js:955:30)","    at Object.Module._extensions..js (internal/modules/cjs/loader.js:991:10)","    at Module.load (internal/modules/cjs/loader.js:811:32)","    at Function.Module._load (internal/modules/cjs/loader.js:723:14)","    at Function.Module.runMain (internal/modules/cjs/loader.js:1043:10)","    at internal/main/run_main_module.js:17:11"]}
?[32mSTART RequestId: b6f39717-746d-1597-9838-3b6472ec8843 Version: $LATEST?[0m
?[32mEND RequestId: b6f39717-746d-1597-9838-3b6472ec8843?[0m
?[32mREPORT RequestId: b6f39717-746d-1597-9838-3b6472ec8843     Init Duration: 237.77 ms        Duration: 3.67 ms       Billed Duration: 100 ms Memory Size: 128 MB     Max Memory Used: 38 MB  ?[0m

{"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'dep'\nRequire stack:\n- /var/task/app.js\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js"}

When I try to start the API locally with sam local start-api, it fails with the same error as above.

I'm thinking that if it weren't for the relative file paths being off during the build phase, I'd be able to have my cake (debugging locally very quickly) and eat it too (run sam build, sam local start-api).

What should I do?

Answer 1

After much frustration and angst, this has been produced: https://github.com/blmille1/aws-sam-layers-template

Enjoy!

Answer 2

I also faced the same issue and eventually came up with an alternative solution to yours. It offers the developer experience you are looking for but avoids the slight inconvenience and maintenance overhead in your solution (aka using the ternary operator for every import that access the shared layer). My proposed solution uses a similar approach as yours but only requires an one-time initialization call per lambda function. Under the hood, it uses module-alias to resolve the dependencies during runtime.

Here's a link to a repository with an example template: https://github.com/dangpg/aws-sam-shared-layers-template

(tl;dr) When using the linked template you get:

• Share common code or dependencies among multiple lambda functions using layers
• Does not use any module bundlers (e.g., Webpack)
• Instead, uses module-alias to resolve dependencies during runtime
• Supports local debugging within VSCode via AWS Toolkit
• Code can be executed outside of AWS sandbox (node app.js)
• Supports unit testing with Jest
• Intellisense/Autocomplete within VSCode
• Compatible with SAM CLI (sam build, sam deploy, sam local invoke, sam local start-api, etc.).
• Can be deployed and run in the cloud as generic lambdas

---

1. Folder structure

+ lambdas
|  + func1
|    - app.js
|    - package.json
|  + func2
|    - app.js
|    - package.json
+ layers
|  + common            // can be any name, I chose common
|    + nodejs          // needs to be nodejs due to how SAM handles layers
|      - index.js      // file for own custom code that you want to share
|      - package.json  // list any dependencies you want to share
- template.yaml

Here's the folder structure I ended up with. Keep in mind though, that it is quite flexible and doesn't require hard rules in order to fulfill possible relative file paths (however, you would need to adapt some files if your structure differs).

If you want to share npm packages among lambda functions, just add them to the package.json within the layer folder:

{
  "name": "common",
  "version": "1.0.0",
  "main": "index.js",
  "dependencies": {
    "lorem-ipsum": "^2.0.4"
  }
}

For completeness, here's the content of index.js:

exports.ping = () => {
  return "pong";
};

2. Template.yaml

[...]
Globals:
  Function:
    Timeout: 3
    Runtime: nodejs14.x
    Environment:
      Variables:
        AWS: true

Resources:
  Func1:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: lambdas/func1/
      Handler: app.lambdaHandler
      Layers:
        - !Ref CommonLayer
  [...]

  Func2:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: lambdas/func2/
      Handler: app.lambdaHandler
      Layers:
        - !Ref CommonLayer
  [...]

  CommonLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      ContentUri: ./layers/common/
      CompatibleRuntimes:
        - nodejs14.x
      RetentionPolicy: Retain

Pretty straightforward, just follow the official example on how to include layers in your lambdas. Like in your solution, add a gloval env variable, so we can differentiate if we are running code within an AWS sandbox or not.

3. Lambda package.json

Add module-alias as dependency and your local common folder as devDependency to each lambda function:

...
  "dependencies": {
    "module-alias": "^2.2.2"
  },
  "devDependencies": {
    "common__internal": "file:../../layers/common/nodejs"  // adapt relative path according to your folder structure
  },
...

We will need the local reference to our common folder later on (e.g. for testing). We add it as devDependency since we only need it for local development and so we don't run into issues when running sam build (since it ignores devDependencies). I chose common__internal as package name, however you are free to choose whatever you like. Make sure to run npm install before doing any local development.

4. Lambda handler

Within your handler source code, before you import any packages from your shared layer, initialize module-alias to do the following:

const moduleAlias = require("module-alias");

moduleAlias.addAlias("@common", (fromPath, request) => {
  if (process.env.AWS && request.startsWith("@common/")) {
    // AWS sandbox and reference to dependency
    return "/opt/nodejs/node_modules";
  }

  if (process.env.AWS) {
    // AWS sandbox and reference to custom shared code
    return "/opt/nodejs";
  }

  if (request.startsWith("@common/")) {
    // Local development and reference to dependency
    return "../../layers/common/nodejs/node_modules";
  }

  // Local development and reference to custom shared code
  return "../../layers/common/nodejs";
});

const { ping } = require("@common");                     // your custom shared code
const { loremIpsum } = require("@common/lorem-ipsum");   // shared dependency

exports.lambdaHandler = async (event, context) => {
  try {
    const response = {
      statusCode: 200,
      body: JSON.stringify({
        message: "hello world",
        ping: ping(),
        lorem: loremIpsum(),
      }),
    };

    return response;
  } catch (err) {
    console.log(err);
    return err;
  }
};

if (require.main === module) {
  (async () => {
    var result = await exports.lambdaHandler(process.argv[1]);
    console.log(result);
  })();
}

You can move the module-alias code part to a separate file and just import that one in the beginning instead (or even publish your own custom package that you can then properly import; this way you can reference it within each of your lambda function and don't have to duplicate code). Again, adjust the relative file paths according to your folder structure. Similar to your approach, it checks for the AWS environment variable and adjusts the import paths accordingly. However, this only has to be done once. After that, all successive imports can just use your defined alias: const { ping } = require("@common"); and const { loremIpsum } = require("@common/lorem-ipsum");. Also here, feel free to define your very own custom logic on how to handle aliases. This is just the solution I came up with that worked for me.

From this point on, you should be able to execute your lambda code either locally through node app.js or through the SAM CLI (sam build, sam local invoke, etc.). However, if you want local testing and intellisense, there is some additional work left.

5. Intellisense

For VSCode, you can just add a jsconfig.json file with the respective path mappings for your defined alias. Point it to the internal devdependency from earlier:

{
  "compilerOptions": {
    "baseUrl": ".",
    "paths": {
      "@common": ["./node_modules/common__internal/index.js"],
      "@common/*": ["./node_modules/common__internal/node_modules/*"]
    }
  }
}

6. Testing

For testing, I personally use Jest. Fortunately, Jest provides the option to provide path mappings too:

// within your lambda package.json
  "jest": {
    "moduleNameMapper": {
      "@common/(.*)": "<rootDir>/node_modules/common__internal/node_modules/$1",
      "@common": "<rootDir>/node_modules/common__internal/"
    }
  }

Final disclaimer

This solution currently only works when using the CommonJS module system. I haven't been able to reproduce the same result when using ES modules (mostly due to the lacking support of module-alias). However, if somebody can come up with a solution using ES modules, I am happy to hear it!

And that's it! Hopefully I didn't leave anything out. Overall, I'm pretty happy with the developer experience this solution is offering. Feel free to look at the linked template repository for more details. I know it's been a bit since your original question but I left it here in the hope that it will maybe help fellow developers too. Cheers

Answer 3

I followed up on your approved answer and I believe I figured out the correct answer (and posting here since I got here via Google and others may wander in here).

1. Organize your module in the following way (my-layers and my-module can be adjusted but the nodejs/node_modules must remain)

+ my-layers
| + my-module
|   + nodejs
|     + node_modules
|       + my-module
|         + index.js
|         + package.json
+ my-serverless
  + template.yaml
  + package.json

I don't know the ideal setup for package.json. Specifying "main": "index.js" was enough for me.

{
  "name": "my-module",
  "version": "1.0.0",
  "main": "index.js"
}

and this is the index.js

exports.ping = () => console.log('pong');

2. In the SAM template lambda layer point to the ../my-layers/

  MyModuleLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: my-module
      ContentUri: ../my-layers/my-module/
      CompatibleRuntimes:
        - nodejs14.x

So far so good. But now - to get the code completion and get rid of the crazy require(process.env.AWS ? '/opt/nodejs/common' : '../../layers/layer1/nodejs/common'); you had in the func2internals.js

3. Add the dependency to the my-serverless

a. either install from CLI:

npm i --save ../my-layers/my-module/nodejs/node_modules/my-module

b. or add in package.json dependencies

  "dependencies": {
    "my-module": "file:../my-layers/my-module/nodejs/node_modules/my-module",
  }

4. Use my-module in your serverless function

var myModule = require('my-module');

exports.handler = async (event) => {
  myModule.ping();
};

That's it. You have code completion and it works on local env and in sam local invoke and in sam local start-api

And dont forgot to exclude my-layers/my-module/nodejs/node_modules from .gitignore :)