I'm working on a project with an AWS Infrastructure. I work with aws-cdk-lib for IaC to simplify the whole process.
The flow is the following:
- User upload photo/video to S3 Bucket
- A lambda is triggered to compress the file with
sharpand store it on another S3 Bucket with a short TTL. - A S3 Notification event trigger another lambda which will upload the S3Object to a Storj Bucket.
So, I need to use uplink-nodejs library to store images and videos on the Storj DCS service.
The issue is that, in order to install uplink-nodejs i need to install make command and copy ./node_modules/uplink-nodejs/* / for uplink command to be available to the system.
But I can't find a way to install all required dependancies and make the required commands.
I've already tried multiple solutions:
- Use
NodejsFunctionbundling options (beforeInstall, beforeBundling, afterBundling). - Create a
.dockerfilein my lambda folder.
The first solution throw an error for su command and sudo command, which prevent me from installing make
The second solution doesn't seems specifically relevant in my situation.
Someone have an idea of what's happening ?
Here's the AWS CDK NestedStack for the ContentService with the lambdas:
import * as cdk from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as path from 'path';
import { Construct } from 'constructs';
import { NodejsFunction } from 'aws-cdk-lib/aws-lambda-nodejs';
import { DockerImage } from 'aws-cdk-lib';
import { LambdaDestination } from 'aws-cdk-lib/aws-s3-notifications';
interface ContentServiceProps extends cdk.NestedStackProps {
main_table: dynamodb.Table;
user_content_bucket: s3.Bucket;
};
export class ContentService extends cdk.NestedStack {
public readonly content_service_handler: NodejsFunction;
public readonly storj_image_lambda: NodejsFunction;
constructor(scope: Construct, id: string, props: ContentServiceProps) {
super(scope, id, props);
this.storj_image_lambda = new NodejsFunction(this, "storj_image_lambda", {
entry: path.join(__dirname, '../../lambda-fns/src/functions/storj_upload/index.ts'),
bundling: {
nodeModules: ["uplink-nodejs", "node-gyp"],
forceDockerBundling: true,
commandHooks: {
beforeInstall(_inputDir, outputDir) {
return [
'export PATH=$PATH:$GOPATH/bin',
'sudo yum -y install make'
]
},
beforeBundling(inputDir, outputDir) {
return [
''
]
},
afterBundling(inputDir, outputDir) {
return [`cp -r ./node_modules/uplink-nodejs /usr`]
}
}
}
});
this.content_service_handler = new NodejsFunction(this, "content_service_handler", {
runtime: lambda.Runtime.NODEJS_16_X,
handler: 'handler',
entry: path.join(__dirname, "../../lambda-fns/src/functions/content/index.ts"),
environment: {
TABLE_NAME: props.main_table.tableName,
USER_CONTENT: props.user_content_bucket.bucketName,
STORJ_LAMBDA: this.storj_image_lambda.functionName
}
});
props.main_table.grantFullAccess(this.content_service_handler);
props.user_content_bucket.grantRead(this.content_service_handler);
props.user_content_bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new LambdaDestination(this.storj_image_lambda));
};
};
And here's the logs I get when I try to deploy the my Stack:
Bundling asset MainStack/content_service/storj_image_lambda/Code/Stage...
asset-output/index.js 1.5kb
⚡ Done in 21ms
bash: sudo: command not found
/Users/thomgeenen/Git/nude_safer_cdk/node_modules/aws-cdk-lib/core/lib/asset-staging.js:2
`),localBundling=options.local?.tryBundle(bundleDir,options),!localBundling){let user;if(options.user)user=options.user;else{const userInfo=os.userInfo();user=userInfo.uid!==-1?`${userInfo.uid}:${userInfo.gid}`:"1000:1000"}options.image.run({command:options.command,user,volumes,environment:options.environment,entrypoint:options.entrypoint,workingDirectory:options.workingDirectory??AssetStaging.BUNDLING_INPUT_DIR,securityOpt:options.securityOpt??""})}}catch(err){const bundleErrorDir=bundleDir+"-error";throw fs.existsSync(bundleErrorDir)&&fs.removeSync(bundleErrorDir),fs.renameSync(bundleDir,bundleErrorDir),new Error(`Failed to bundle asset ${this.node.path}, bundle output is located at ${bundleErrorDir}: ${err}`)}if(fs_1.FileSystem.isEmpty(bundleDir)){const outputDir=localBundling?bundleDir:AssetStaging.BUNDLING_OUTPUT_DIR;throw new Error(`Bundling did not produce any output. Check that content is written to ${outputDir}.`)}}calculateHash(hashType,bundling,outputDir){if(hashType==assets_1.AssetHashType.CUSTOM||hashType==assets_1.AssetHashType.SOURCE&&bundling){const hash=crypto.createHash("sha256");return hash.update(this.customSourceFingerprint??fs_1.FileSystem.fingerprint(this.sourcePath,this.fingerprintOptions)),bundling&&hash.update(JSON.stringify(bundling)),hash.digest("hex")}switch(hashType){case assets_1.AssetHashType.SOURCE:return fs_1.FileSystem.fingerprint(this.sourcePath,this.fingerprintOptions);case assets_1.AssetHashType.BUNDLE:case assets_1.AssetHashType.OUTPUT:if(!outputDir)throw new Error(`Cannot use \`${hashType}\` hash type when \`bundling\` is not specified.`);return fs_1.FileSystem.fingerprint(outputDir,this.fingerprintOptions);default:throw new Error("Unknown asset hash type.")}}}exports.AssetStaging=AssetStaging,_a=JSII_RTTI_SYMBOL_1,AssetStaging[_a]={fqn:"aws-cdk-lib.AssetStaging",version:"2.53.0"},AssetStaging.BUNDLING_INPUT_DIR="/asset-input",AssetStaging.BUNDLING_OUTPUT_DIR="/asset-output",AssetStaging.assetCache=new cache_1.Cache;function renderAssetFilename(assetHash,extension=""){return`asset.${assetHash}${extension}`}function determineHashType(assetHashType,customSourceFingerprint){const hashType=customSourceFingerprint?assetHashType??assets_1.AssetHashType.CUSTOM:assetHashType??assets_1.AssetHashType.SOURCE;if(customSourceFingerprint&&hashType!==assets_1.AssetHashType.CUSTOM)throw new Error(`Cannot specify \`${assetHashType}\` for \`assetHashType\` when \`assetHash\` is specified. Use \`CUSTOM\` or leave \`undefined\`.`);if(hashType===assets_1.AssetHashType.CUSTOM&&!customSourceFingerprint)throw new Error("`assetHash` must be specified when `assetHashType` is set to `AssetHashType.CUSTOM`.");return hashType}function calculateCacheKey(props){return crypto.createHash("sha256").update(JSON.stringify(sortObject(props))).digest("hex")}function sortObject(object){if(typeof object!="object"||object instanceof Array)return object;const ret={};for(const key of Object.keys(object).sort())ret[key]=sortObject(object[key]);return ret}function singleArchiveFile(directory){if(!fs.existsSync(directory))throw new Error(`Directory ${directory} does not exist.`);if(!fs.statSync(directory).isDirectory())throw new Error(`${directory} is not a directory.`);const content=fs.readdirSync(directory);if(content.length===1){const file=path.join(directory,content[0]),extension=getExtension(content[0]).toLowerCase();if(fs.statSync(file).isFile()&&ARCHIVE_EXTENSIONS.includes(extension))return file}}function determineBundledAsset(bundleDir,outputType){const archiveFile=singleArchiveFile(bundleDir);switch(outputType===bundling_1.BundlingOutput.AUTO_DISCOVER&&(outputType=archiveFile?bundling_1.BundlingOutput.ARCHIVED:bundling_1.BundlingOutput.NOT_ARCHIVED),outputType){case bundling_1.BundlingOutput.NOT_ARCHIVED:return{path:bundleDir,packaging:assets_1.FileAssetPackaging.ZIP_DIRECTORY};case bundling_1.BundlingOutput.ARCHIVED:if(!archiveFile)throw new Error("Bundling output directory is expected to include only a single archive file when `output` is set to `ARCHIVED`");return{path:archiveFile,packaging:assets_1.FileAssetPackaging.FILE,extension:getExtension(archiveFile)}}}function getExtension(source){for(const ext of ARCHIVE_EXTENSIONS)if(source.toLowerCase().endsWith(ext))return ext;return path.extname(source)}
^
Error: Failed to bundle asset MainStack/content_service/storj_image_lambda/Code/Stage, bundle output is located at /Users/thomgeenen/Git/nude_safer_cdk/cdk.out/bundling-temp-4e95a752d2aca74dd84aa402102b31591eb7ee9ec0a94b306360353a61e924f0-error: Error: docker exited with status 127
at AssetStaging.bundle (/Users/thomgeenen/Git/nude_safer_cdk/node_modules/aws-cdk-lib/core/lib/asset-staging.js:2:614)
at AssetStaging.stageByBundling (/Users/thomgeenen/Git/nude_safer_cdk/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:4506)
at stageThisAsset (/Users/thomgeenen/Git/nude_safer_cdk/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:1867)
at Cache.obtain (/Users/thomgeenen/Git/nude_safer_cdk/node_modules/aws-cdk-lib/core/lib/private/cache.js:1:242)
at new AssetStaging (/Users/thomgeenen/Git/nude_safer_cdk/node_modules/aws-cdk-lib/core/lib/asset-staging.js:1:2262)
at new Asset (/Users/thomgeenen/Git/nude_safer_cdk/node_modules/aws-cdk-lib/aws-s3-assets/lib/asset.js:1:736)
at AssetCode.bind (/Users/thomgeenen/Git/nude_safer_cdk/node_modules/aws-cdk-lib/aws-lambda/lib/code.js:1:4628)
at new Function (/Users/thomgeenen/Git/nude_safer_cdk/node_modules/aws-cdk-lib/aws-lambda/lib/function.js:1:2803)
at new NodejsFunction (/Users/thomgeenen/Git/nude_safer_cdk/node_modules/aws-cdk-lib/aws-lambda-nodejs/lib/function.js:1:1171)
at new ContentService (/Users/thomgeenen/Git/nude_safer_cdk/lib/services/content/index.ts:24:35)
Detected file changes during deployment. Invoking 'cdk deploy' again
[+] Building 0.5s (13/13) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for public.ecr.aws/sam/build-nodejs14.x:latest 0.4s
=> [1/9] FROM public.ecr.aws/sam/build-nodejs14.x@sha256:cfe32c14b97a6d5d7128f4e110c6d11b80bf89c0cca8a3493879f8c587627655 0.0s
=> CACHED [2/9] RUN npm install --global yarn@1.22.5 0.0s
=> CACHED [3/9] RUN npm install --global pnpm 0.0s
=> CACHED [4/9] RUN npm install --global typescript 0.0s
=> CACHED [5/9] RUN npm install --global --unsafe-perm=true esbuild@0 0.0s
=> CACHED [6/9] RUN mkdir /tmp/npm-cache && chmod -R 777 /tmp/npm-cache && npm config --global set cache /tmp/npm-cache 0.0s
=> CACHED [7/9] RUN mkdir /tmp/yarn-cache && chmod -R 777 /tmp/yarn-cache && yarn config set cache-folder /tmp/yarn-cache 0.0s
=> CACHED [8/9] RUN npm config --global set update-notifier false 0.0s
=> CACHED [9/9] RUN /sbin/useradd -u 1000 user && chmod 711 / 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:c83389551577052395325ed8a0fb51c59639344b8c92baa1656956f23c765d18 0.0s
=> => naming to docker.io/library/cdk-7ef608663d730a301f1ab98604e57fd96273751c63db25a1a75f1390f462c655 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
