AWS Beanstalk replacing instance upon sending no data

Question

We have a single on-demand instance of a staging version of a Node app hosted on AWS Beanstalk.

Around 430am EST on a Friday (where there would be very little load,traffic)

the environment had the following event "Environment health has transitioned from Ok to No Data. None of the instances are sending data."

This was then succeeded by "Added instance [i-SOME_ID] to your environment."

and then "Removed instance [i-ANOTHER_ID] from your environment."

and then "Environment health has transitioned from No Data to Severe. Command is executing on all instances."

and finally, "Environment health has transitioned from Severe to Ok."

There were no recent updates to the app's code itself and I have managed updates disabled. I don't see anything else in the logs that would indicate a random crash.

How can I deduce what happened here? Also, is there a way to have EB not replace instances automatically like this? Every time an instance is replaced, we need to change settings on our end because the EIP is updates (CNAME, etc..)

Edit: I pulled the 'full logs', but the issue is that the logs only show everything after the instance got rebuilt (and not before whatever caused the 'Environment health has transitioned from Ok to No Data. None of the instances are sending data.')

Below is my YAML conf

    ApplicationName: 'MY_APP_NAME'
DateUpdated: 2021-10-15 15:23:16+00:00
EnvironmentName: MY_ENV_NAME-env
PlatformArn: arn:aws:elasticbeanstalk:us-east-2::platform/Node.js running on 64bit
  Amazon Linux/4.17.6
settings:
  aws:autoscaling:asg:
    Availability Zones: Any
    Cooldown: '360'
    Custom Availability Zones: ''
    EnableCapacityRebalancing: 'false'
    MaxSize: '1'
    MinSize: '1'
  aws:autoscaling:launchconfiguration:
    BlockDeviceMappings: null
    EC2KeyName: MY_KEY_NAME
    IamInstanceProfile: aws-elasticbeanstalk-ec2-role
    ImageId: ami-MY_AMI_ID
    InstanceType: t2.micro
    MonitoringInterval: 5 minute
    RootVolumeIOPS: null
    RootVolumeSize: null
    RootVolumeThroughput: null
    RootVolumeType: null
    SSHSourceRestriction: tcp,22,22,0.0.0.0/0
    SecurityGroups: MY_SGs
  aws:autoscaling:updatepolicy:rollingupdate:
    MaxBatchSize: null
    MinInstancesInService: null
    PauseTime: null
    RollingUpdateEnabled: 'false'
    RollingUpdateType: Time
    Timeout: PT30M
  aws:ec2:instances:
    EnableSpot: 'false'
    InstanceTypes: t2.micro, t2.small
    SpotFleetOnDemandAboveBasePercentage: '0'
    SpotFleetOnDemandBase: '0'
    SpotMaxPrice: null
  aws:ec2:vpc:
    AssociatePublicIpAddress: 'false'
    ELBScheme: public
    ELBSubnets: subnet-MY_SUBNET_ID
    Subnets: subnet-MY_SUBNET_ID
    VPCId: vpc-MY_VPC_ID
  aws:elasticbeanstalk:application:
    Application Healthcheck URL: ''
  aws:elasticbeanstalk:cloudwatch:logs:
    DeleteOnTerminate: 'true'
    RetentionInDays: '7'
    StreamLogs: 'true'
  aws:elasticbeanstalk:cloudwatch:logs:health:
    DeleteOnTerminate: 'false'
    HealthStreamingEnabled: 'false'
    RetentionInDays: '7'
  aws:elasticbeanstalk:command:
    BatchSize: '100'
    BatchSizeType: Percentage
    DeploymentPolicy: AllAtOnce
    IgnoreHealthCheck: 'false'
    Timeout: '600'
  aws:elasticbeanstalk:container:nodejs:
    GzipCompression: 'true'
    NodeCommand: null
    NodeVersion: 12.22.1
    ProxyServer: none
  aws:elasticbeanstalk:control:
    DefaultSSHPort: '22'
    LaunchTimeout: '0'
    LaunchType: Migration
    RollbackLaunchOnFailure: 'false'
  aws:elasticbeanstalk:environment:
    EnvironmentType: SingleInstance
    ExternalExtensionsS3Bucket: null
    ExternalExtensionsS3Key: null
    ServiceRole: arn:aws:iam::MY_IAM_ID:role/aws-elasticbeanstalk-service-role
  aws:elasticbeanstalk:healthreporting:system:
    ConfigDocument: '{"Version":1,"CloudWatchMetrics":{"Instance":{"RootFilesystemUtil":null,"CPUIrq":null,"LoadAverage5min":null,"ApplicationRequests5xx":null,"ApplicationRequests4xx":null,"CPUUser":null,"LoadAverage1min":null,"ApplicationLatencyP50":null,"CPUIdle":null,"InstanceHealth":null,"ApplicationLatencyP95":null,"ApplicationLatencyP85":null,"ApplicationLatencyP90":null,"CPUSystem":null,"ApplicationLatencyP75":null,"CPUSoftirq":null,"ApplicationLatencyP10":null,"ApplicationLatencyP99":null,"ApplicationRequestsTotal":null,"ApplicationLatencyP99.9":null,"ApplicationRequests3xx":null,"ApplicationRequests2xx":null,"CPUIowait":null,"CPUNice":null},"Environment":{"InstancesSevere":null,"InstancesDegraded":null,"ApplicationRequests5xx":null,"ApplicationRequests4xx":null,"ApplicationLatencyP50":null,"ApplicationLatencyP95":null,"ApplicationLatencyP85":null,"InstancesUnknown":null,"ApplicationLatencyP90":null,"InstancesInfo":null,"InstancesPending":null,"ApplicationLatencyP75":null,"ApplicationLatencyP10":null,"ApplicationLatencyP99":null,"ApplicationRequestsTotal":null,"InstancesNoData":null,"ApplicationLatencyP99.9":null,"ApplicationRequests3xx":null,"ApplicationRequests2xx":null,"InstancesOk":null,"InstancesWarning":null}},"Rules":{"Environment":{"ELB":{"ELBRequests4xx":{"Enabled":true}},"Application":{"ApplicationRequests4xx":{"Enabled":true}}}}}'
    EnhancedHealthAuthEnabled: 'false'
    HealthCheckSuccessThreshold: Ok
    SystemType: enhanced
  aws:elasticbeanstalk:hostmanager:
    LogPublicationControl: 'false'
  aws:elasticbeanstalk:managedactions:
    ManagedActionsEnabled: 'false'
    PreferredStartTime: Mon:04:00
  aws:elasticbeanstalk:managedactions:platformupdate:
    InstanceRefreshEnabled: 'false'
    UpdateLevel: minor
  aws:elasticbeanstalk:monitoring:
    Automatically Terminate Unhealthy Instances: 'true'
  aws:elasticbeanstalk:sns:topics:
    Notification Endpoint: MY_EMAIL
    Notification Protocol: email
    Notification Topic ARN: arn:aws:sns:us-east-2:MY_ARN_ID:ElasticBeanstalkNotifications-Environment-22-env
    Notification Topic Name: null
  aws:elasticbeanstalk:xray:
    XRayEnabled: 'false'
  aws:rds:dbinstance:
    HasCoupledDatabase: 'false'

Can you provide some details on the configuration? Instance type, root volume, any auto scaling being used (capacity config section), are you using enhanced health monitoring? Regarding your second question (instance replacement) - what selection have you made for "deployment policy" and "Rolling update type" (Rolling updates and deployments config section)? Unfortunately there are many ways to encounter the scenario you are experiencing and it can be triggered by many details unique to a given app/environment/config implementation. — dusthaines, Oct 13 '21 at 22:38
It's a single instance type and it says the fleet can either accept t2.micro or t2.small. It says capacity rebalancing is disabled (so I think that means auto-scaling is). It is using enhanced health monitoring. The deployment policy is "all at once" and Rolling updates are disabled. — user2402616, Oct 13 '21 at 23:35
Thank you for sharing, all helpful details. Best initial guess is this is probably related to enhanced health. Can you confirm that `nginx` is being used as the proxy server (it would be by default) and also share details on any nginx config modification you might have in place (if applicable). On the platform side please confirm if you're using Node via Amazon Linux or Amazon Linux 2 platform. Lastly on the app side, please confirm if you are using an application packaging framework (like Docker). — dusthaines, Oct 14 '21 at 17:42
Hi! I'm actually running a deprecated Node platform. "Node.js running on 64bit Amazon Linux/4.17.6" I am not using NGINX or any reverse-proxy. The app side is just a vanilla Node project (nothing container based). — user2402616, Oct 14 '21 at 17:54
Good to know but also means a common issue is probably not your issue. I expected you would be on Linux 1 but not that you would be running without a proxy server.`nginx` configs/logs and how they tie into advanced health, are a common issue. Still a way to figure this out, just a lot more places to check! Next best step is to share your instance config. If you use the `eb cli`, type `eb config` in your terminal, then `^O` to write out the details to a `yml` file. Remove any private entries (like `EC2KeyName`, `ServiceRole`, anything with `iam` in it). Post for review, we'll go from there. — dusthaines, Oct 15 '21 at 17:15
I have it installed but get `ERROR: This directory has not been set up with the EB CLI You must first run "eb init".` when running `eb init` I feel like it's trying to create a new Env (which I don't want) — user2402616, Oct 15 '21 at 17:28
Fastest way without the CLI is probably to navigate to the config section for your instance, toggle the switch at top right to "show table view". Then click-drag the select and copy/paste the contents to a text file, tidy up and remove private info, then add to your post. Unfortunately there is additional info in the full config that is not exposed in the web tools, so might need to use the CLI eventually but we can start with this. — dusthaines, Oct 15 '21 at 17:34
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/238178/discussion-between-user2402616-and-dusthaines). — user2402616, Oct 15 '21 at 17:38
When you run `eb init`, it will prompt you first to select your region, then it will ask you to select which application to link to, and you can make the relevant selection. I believe you must already have your global `config` and `credentials`set up in `~.aws` folder though. If you've never used `eb cli` those steps might be necessary as well. — dusthaines, Oct 15 '21 at 17:39

AWS Beanstalk replacing instance upon sending no data

0 Answers0