4

I would like to ask for help about Elastic Beanstalk error:

Environment health has transitioned from Ok to Severe. 81.8 % of the requests are erroring with HTTP 4xx.

I read some articles here and I followed the solution with WAF, so I created ACL which is assigned to our CloudFront, then I created the rule, which blocks all requests which contain word HEAD in HTTP Method. When I try to send HEAD request from postman, then it works like I want (I receive error 403), but unfortunately the error still exists and I see a lot of HEAD requests in the apache logs every day.

List of requests:

[01/Aug/2017:07:42:09 +0000] "HEAD /mysql/dbadmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:11 +0000] "HEAD /mysql/mysqlmanager/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:11 +0000] "HEAD /phpMyadmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:11 +0000] "HEAD /phpmyAdmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:12 +0000] "HEAD /phpmyadmin3/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:13 +0000] "HEAD /2phpmyadmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:13 +0000] "HEAD /phppma/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:14 +0000] "HEAD /shopdb/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:15 +0000] "HEAD /program/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:15 +0000] "HEAD /dbadmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:16 +0000] "HEAD /db/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:16 +0000] "HEAD /mysql/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:17 +0000] "HEAD /db/phpmyadmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:17 +0000] "HEAD /sqlmanager/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:18 +0000] "HEAD /php-myadmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:19 +0000] "HEAD /mysqladmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:19 +0000] "HEAD /admin/phpmyadmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:20 +0000] "HEAD /admin/sysadmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:20 +0000] "HEAD /admin/db/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:21 +0000] "HEAD /admin/pMA/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:22 +0000] "HEAD /mysql/db/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:23 +0000] "HEAD /mysql/pMA/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:24 +0000] "HEAD /sql/php-myadmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:24 +0000] "HEAD /sql/sql/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:25 +0000] "HEAD /sql/webadmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:26 +0000] "HEAD /sql/websql/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:30 +0000] "HEAD /sql/sqladmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:30 +0000] "HEAD /sql/phpmyadmin2/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:31 +0000] "HEAD /sql/phpMyAdmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:38 +0000] "HEAD /db/webadmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:43 +0000] "HEAD /db/websql/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:49 +0000] "HEAD /db/dbadmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:49 +0000] "HEAD /db/phpmyadmin3/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:51 +0000] "HEAD /db/phpMyAdmin-3/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:52 +0000] "HEAD /administrator/phpMyAdmin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:52 +0000] "HEAD /administrator/web/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:54 +0000] "HEAD /administrator/PMA/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:54 +0000] "HEAD /phpMyAdmin2/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:55 +0000] "HEAD /phpMyAdmin4/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:55 +0000] "HEAD /php-my-admin/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:56 +0000] "HEAD /PMA2012/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:56 +0000] "HEAD /PMA2014/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:57 +0000] "HEAD /PMA2016/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:57 +0000] "HEAD /PMA2018/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:58 +0000] "HEAD /pma2012/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:42:59 +0000] "HEAD /pma2014/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:43:00 +0000] "HEAD /pma2016/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:43:01 +0000] "HEAD /pma2018/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:43:01 +0000] "HEAD /phpmyadmin2012/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:43:02 +0000] "HEAD /phpmyadmin2014/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:43:02 +0000] "HEAD /phpmyadmin2016/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

[01/Aug/2017:07:43:04 +0000] "HEAD /phpmyadmin2018/ HTTP/1.1" 404 260 "-" "Mozilla/5.0 Jorgee"

Thanks for the help.

David Roušar
  • 81
  • 1
  • 6

3 Answers3

3

I contacted direct AWS Support and this is the solution which they provided to me:

I looked at the logs that you posted in case, I found that the agent is Jorgee, which is a common malware agent. I came across the blog regarding to this agent [1], though it is not official one but got insights of it.

A daemon named "healthd" in Elastic Beanstalk environment instances monitor health by watching special log files. If the agent find lots of 4xx in this file, the environment go to Severe state.

$ sudo tail /var/log/nginx/healthd/application.log.2017-08-21-07 1503299631.249"/asdf"404"0.075"0.075"- 1503299631.379"/asdf"404"0.002"0.002"-

I see you have environments launched with the solution stack "64bit Amazon Linux 2017.03 v2.7.2 running Docker 17.03.1-ce", thus I would like to provide a workaround of this issue for this solution stack.

In the solution stack "64bit Amazon Linux 2017.03 v2.7.2 running Docker 17.03.1-ce", log format above is defined in "/etc/nginx/nginx.conf", and enabled in "/etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf".

Therefore, you could configure nginx in your environments to ignore requests which HTTP status is 404 or 403. Please try to add following config file under .ebextensions directory of your application source code bundle.

.ebextensions/healthd_ignore_4xx.config

   files:
   "/etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf":
   mode: "000644"
   owner: root
   group: root
   content: |
     # modification No.1
      map $status $logflag {
          404 0;
          403 0;
          default 1;
      }

      map $http_upgrade $connection_upgrade {
          default        "upgrade";
          ""            "";
      }

      server {
          listen 80;

          gzip on;
              gzip_comp_level 4;
              gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

          if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
              set $year $1;
              set $month $2;
              set $day $3;
              set $hour $4;
          }

          # modification No.2
          # access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;
            access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd if=$logflag;

          access_log    /var/log/nginx/access.log;

          location / {
              proxy_pass            http://docker;
              proxy_http_version    1.1;

              proxy_set_header    Connection            $connection_upgrade;
              proxy_set_header    Upgrade                $http_upgrade;
              proxy_set_header    Host                $host;
              proxy_set_header    X-Real-IP            $remote_addr;
              proxy_set_header    X-Forwarded-For        $proxy_add_x_forwarded_for;
          }
      }

This config will replace default /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy.conf file with the content you defined. Modifications I made are:

  • No.1: added map directive which maps from $status to $logflag. when the request is 404 or 403, set $logflag to 0. set 1 for other status.
  • No.2: added if=$logflag in access_log [2] directive. Write to healthd monitoring logs only when the HTTP status is not 404 or 403.

After you deploy the new version application with ebextensions config above, your environment status will not be affected by invalid 404 or 403 requests.

References [1]: http://www.skepticism.us/2015/05/new-malware-user-agent-value-jorgee/ [2]: http://nginx.org/en/docs/stream/ngx_stream_log_module.html#access_log

David Roušar
  • 81
  • 1
  • 6
2

For me, I didn't have a Response for the root(/) so by simply adding a dummy page in spring-boot my problem with ELB went away.

@GetMapping("/")
@ResponseBody
public String sayHello() {
    return "hello";
}
Jghorton14
  • 724
  • 1
  • 8
  • 25
0

To solve the issue,

I changed the elasticbeans load balancer to application level one and enabled WAF integration.

In WAF, I defined following rules to prevent malware requests.

 URI contains: "/pma" after converting to lowercase.
 URI contains: "/sql" after converting to lowercase.
 URI contains: "/admin" after converting to lowercase.
 URI ends with: "php" after converting to lowercase.
 URI contains: "/mysql" after converting to lowercase.
 URI contains: "/db" after converting to lowercase.
 URI contains: "/2phpmyadmin/ " after converting to lowercase.
 URI contains: "/shopdb/ " after converting to lowercase.
 URI contains: "/php" after converting to lowercase.
Eren Yagdiran
  • 338
  • 2
  • 12