5

codedeploy failed after few deployments(php application) with error

"Cannot allocate memory - su" but in instance its sowing 51% memory in use.

What could be reason for it?

Error.
 InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: Errno::ENOMEM - Cannot allocate memory - su - /usr/share/ruby/open3.rb:211:in `spawn'
/usr/share/ruby/open3.rb:211:in `popen_run'
/usr/share/ruby/open3.rb:99:in `popen3'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:141:in `execute_script'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:115:in `block (2 levels) in execute'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:103:in `each'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:103:in `block in execute'


[centos@ip-10-196-21-200 codedeploy-agent]$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/xvda1      41926416 21195568  20730848  51% /
devtmpfs          487892        0    487892   0% /dev
tmpfs             507480        0    507480   0% /dev/shm
tmpfs             507480    38336    469144   8% /run
tmpfs             507480        0    507480   0% /sys/fs/cgroup
tmpfs             101496        0    101496   0% /run/user/1000
Andrew Lohr
  • 5,380
  • 1
  • 26
  • 38
Khushboo
  • 51
  • 1
  • 4

3 Answers3

5

It's likely this memory leak issue in the CodeDeploy agent: https://github.com/aws/aws-codedeploy-agent/issues/32

Daniel Silva
  • 141
  • 3
  • 5
5

There is no official fix for this yet as of writing this.

The temporary fix for Ubuntu is as follows.

In your appspec.yml file add the following:

  ValidateService:
    - location: scripts/validate_service.sh
      timeout: 30
      runas: root

In your scripts folder, create a validate_service.sh file, inside the file add the following:

#!/bin/bash
echo "service codedeploy-agent restart" | at -M now + 2 minute;

This will restart the service after 2 minutes after each deployment.

That is the current working fix for it, AWS seems to be working on the problem. The RubyZip seems to be using all of the memory for the server and not releasing it.

Read more here: https://github.com/aws/aws-codedeploy-agent/issues/32

Drmzindec
  • 804
  • 2
  • 15
  • 28
1

There's another solution if you are storing the deployment package in S3, but you will have to maintain two packages. Given that memory allocation issue occurs when CodeDeploy agent tries to extract a large archive, the workaround is to deploy a small archive which contains the appspec.yml and a bash or a powershell script.

Within the bash/powershell script you can specify all the required steps to download the package from S3 and extract it to the correct location. Since extraction is not handled by the CodeDeploy agent this time, you will not run into memory allocation issues.

Basically you will have two packages in s3:

  1. full_deployment_package.zip
  2. deploy_package_downloader.zip

full_deployment_package.zip should contain all the application code, etc. And deploy_package_downloader.zip should only contain appspec.yml and a script that downloads and extracts full_deployment_package.zip.

Your appspec.yml can look something like this.

version: 0.0
os: windows
files:
  - source: .
    destination: C:\temp 
hooks: 
  AfterInstall:
    - location: get-full-package.ps1
      timeout: 600

And get-full-package.ps1 can look something like this.

aws s3 cp s3://bucket-name/path/to/full_deployment_package.zip C:\temp\full_deployment_package.zip

Expand-Archive -Path C:\temp\full_deployment_package.zip -DestinationPath C:\path\to\directory

Finally, you should be deploying package_downloader.zip via CodeDeploy and since it will be small in size, it will be deployed without an issue and the script in AfterInstall hook will take care of downloading the actual deployment package.

nonbeing
  • 6,907
  • 6
  • 36
  • 46
Anushka
  • 11
  • 2