Django ALLOWED_HOSTS with ELB HealthCheck

Question

I have a django application deployed on Elastic Beanstalk. The HealthCheck for my app keeps failing because the IP of the ELB HealthCheck is not included in my ALLOWED_HOSTS settings variable.

How can I modify ALLOWED_HOSTS to make the HealthCheck pass? I would just pass in the explicit IP address, but I believe that this changes, so whenever the IP changes the check would fail again until I add the new IP.

Answer 1

Here is another solution using Django Middleware.

Django's django.middleware.common.CommonMiddleware calls request.get_host(), which validates the request with ALLOWED_HOSTS. If you simply want to check that the application is running, you can create a middleware like this.

from django.http import HttpResponse


class HealthCheckMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        if request.path == '/health':
            return HttpResponse('ok')
        return self.get_response(request)

And put your HealthCheckMiddleware in front of CommonMiddleware in settings.py

MIDDLEWARE = [
    'yourdjangoapp.middleware.HealthCheckMiddleware',
    ......
    'django.middleware.common.CommonMiddleware',
    ......
]

And your application will always respond to path /health with ok as long as your app is running regardless of any configurations.

Answer 2

Your EC2 instance can query metadata about itself, including its IP address which is available at: http://169.254.169.254/latest/meta-data/local-ipv4.

You can test this by ssh-ing into your EC2 instance and running:

curl http://169.254.169.254/latest/meta-data/local-ipv4

So, in your configuration, you can do something like:

import requests

ALLOWED_HOSTS = ['.yourdomain.com', ]
try:
    EC2_IP = requests.get('http://169.254.169.254/latest/meta-data/local-ipv4').text
    ALLOWED_HOSTS.append(EC2_IP)
except requests.exceptions.RequestException:
    pass

Obviously you can replace requests with urllib if you don't want the dependency.

Answer 3

Solution

The solution that worked for me was to simply install the django-ebhealthcheck library. After installing it, just add ebhealthcheck.apps.EBHealthCheckConfig to your INSTALLED_APPS.

From django-ebhealthcheck GitHub:

By default, Elastic Beanstalk's health check system uses the public IP of each load balanced instance as the request's host header when making a request. Unless added to ALLOWED_HOSTS, this causes Django to return a 400 Bad Request and a failed health check.

This app dynamically adds your instance's public IP address to Django's ALLOWED_HOSTS setting to permit health checks to succeed. This happens upon application start.

Version 2.0.0 and higher supports IMDSv2. If you are using v1 and cannot upgrade, use version 1 of this library instead (pip install django-ebhealthcheck<2.0.0).

---

Installation

1. pip install django-ebhealthcheck

2. Add ebhealthcheck.apps.EBHealthCheckConfig to your INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    'ebhealthcheck.apps.EBHealthCheckConfig',
    ...
]

Answer 4

To expand on the answer provided by dfrdmn:

While this answer works well in most cases, it has a couple small potential problems.

AWS ELB Network Load Balancers

First, if you are using an ELB network load balancer, this method won't work with its HTTP health checks because the load balancer sends the IP address of the load balancer in the HTTP host header. From the AWS docs:

The HTTP host header in the health check request contains the IP address of the load balancer node and the listener port, not the IP address of the target and the health check port. If you are mapping incoming requests by host header, you must ensure that health checks match any HTTP host header. Another option is to add a separate HTTP service on a different port and configure the target group to use that port for health checks instead. Alternatively, consider using TCP health checks.

So, adding your instance (target group) IP to your ALLOWED_HOSTS will not work. As stated, you could use TCP health checks, or you could use the middleware approach described in another answer.

Metadata endpoint is throttled

Second, because the metadata endpoint limits number of concurrent connections and throttles requests, you may encounter issues in some cases.

Your Django settings.py file is executed for every process and any time processes need to restart. This is important if your webserver is configured to use multiple processes, such as when using gunicorn workers, as is commonly configured to properly take full advantage of system CPU resources.

This means that, with enough processes, your settings.py file will be executed many times, sending many concurrent requests to the metadata endpoint and your processes could fail to start. Further, on subsequent process restarts, the throttling will exacerbate the throttling problem. In some circumstances, this can cause your application to grind to a halt or have fewer processes running than intended.

To get around this, you could do a few things:

1. Obtain the IP address before starting your server and set the IP address as an environment variable, then read the environment variable to add it to your allowed hosts.

$ export ALLOWED_HOST_EC2_PRIVATE_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)
$ gunicorn -w 10 ... myapp:app

# settings.py

ALLOWED_HOSTS = ['myhost.tld', ]

if os.getenv('ALLOWED_HOST_EC2_PRIVATE_IP'):
    ALLOWED_HOSTS.append(os.environ['ALLOWED_HOST_EC2_PRIVATE_IP'])

You may yet still encounter throttling issues with the metadata endpoint if many applications or other services utilize the instance's metadata at the same time.

2. For services running in containers on ECS you can use the container metadata file

You can do this safely within the settings.py because there is no throttling or rate limit for accessing this file. This also avoids your application potentially interfering with other services that need the instance's metadata endpoint.

# settings.py
import os
import json

ALLOWED_HOSTS = ['myhost.tld', ]

if os.getenv('ECS_CONTAINER_METADATA_FILE'):
    metadata_file_path = os.environ['ECS_CONTAINER_METADATA_FILE']
    with open(metadata_file_path) as f:
        metadata = json.load(f)
    private_ip = metadata["HostPrivateIPv4Address"]
    ALLOWED_HOSTS.append(private_ip)

You could also combine the first approach with the metadata file, in your container's ENTRYPOINT.

#!/usr/bin/env bash
# docker-entrypoint.sh
export ALLOWED_HOST_EC2_PRIVATE_IP=$(jq -r .HostPrivateIPv4Address $ECS_CONTAINER_METADATA_FILE)

exec "$@"

FROM myapplication
COPY docker-entrypoint.sh /docker-entrypoint.sh
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["gunicorn", "whatever"]

Answer 5

I like Watt Imsuri's answer above, and I think I'm going to use this approach in all my projects so I turned it into a pip package for Django.

pip install django-easy-health-check

Add the health check middleware to Django settings before django.middleware.common.CommonMiddleware:

MIDDLEWARE = [
    ...,
    'easy_health_check.middleware.HealthCheckMiddleware',
    'django.middleware.common.CommonMiddleware',
    ...
]

By default, the health check url will be available at "example.com/healthcheck/".

You can also customize and overwrite the default settings by including the following in your project's settings.py:

DJANGO_EASY_HEALTH_CHECK = {
    "PATH": "/healthcheck/",
    "RETURN_STATUS_CODE": 200,
    "RETURN_BYTE_DATA": "",
    "RETURN_HEADERS": None
}

In production, you may also want to set the following Django settings:

ALLOWED_HOSTS = ["example.com"]
SECURE_SSL_REDIRECT = True
SECURE_REDIRECT_EXEMPT = [r'^healthcheck/$']

For more info, check out the git.

Answer 6

The other solution doesn't answer the question because it doesn't take into consideration all the various tools AWS has (ELB, etc). What we ended up doing (since we use nginx + uwsgi) is we set the header to something valid when the user sends a request.

As documented on this page:

https://opbeat.com/blog/posts/the-new-django-allowed_hosts-with-elb-uwsgi-and-nginx/

We put our nginx configuration as below:

set $my_host $host;
if ($host ~ "\d+\.\d+\.\d+\.\d+") {
  set $my_host "example.com";
}

location / {
  uwsgi_pass unix:///tmp/mysite.com.sock;
  uwsgi_param HTTP_HOST $my_host;
  include uwsgi_params;
}

The key here is to put a valid value for $my_host as per your ALLOWED_HOSTS.

Unfortunately there is no "perfect" solution without increasing overhead. Some configurations will require you to add IP addresses all the time to the ALLOWED_HOSTS but this solution would generally work with the least over head.