With Elastic Beanstalk, can I determine programmatically if I'm on the leader node?

Question

I have some housekeeping tasks within an Elastic Beanstalk Java application running on Tomcat, and I need to run them every so often. I want these tasks run only on the leader node (or, more correctly, on a single node, but the leader seems like an obvious choice).

I was looking at running cron jobs within Elastic Beanstalk, but it feels like this should be more straightforward than what I've come up with. Ideally, I'd like one of these two options within my web app:

Some way of testing within the current JRE whether or not this server is the leader node
Some some way to hit a specific URL (wget?) to trigger the task, but also restrict that URL to requests from localhost.

Suggestions?

As for #2, look for Worker Environment Tiersin AWS EB. They install an extra app called SQSD which polls for SQS messages and forwards it locally. See http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html for more details — aldrinleal, Feb 24 '14 at 11:13

aldrinleal · Accepted Answer · 2014-02-24T09:49:02.227

It is not possible, by design (leaders are only assigned during deployment, and not needed on other contexts). However, you can tweak and use the EC2 Metadata for this exact purpose.

Here's an working example about how to achieve this result (original source). Once you call getLeader, it will find - or assign - an instance to be set as a leader:

package br.com.ingenieux.resource;

import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;

import javax.inject.Inject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;

import org.apache.commons.io.IOUtils;

import com.amazonaws.services.ec2.AmazonEC2;
import com.amazonaws.services.ec2.model.CreateTagsRequest;
import com.amazonaws.services.ec2.model.DeleteTagsRequest;
import com.amazonaws.services.ec2.model.DescribeInstancesRequest;
import com.amazonaws.services.ec2.model.Filter;
import com.amazonaws.services.ec2.model.Instance;
import com.amazonaws.services.ec2.model.Reservation;
import com.amazonaws.services.ec2.model.Tag;
import com.amazonaws.services.elasticbeanstalk.AWSElasticBeanstalk;
import com.amazonaws.services.elasticbeanstalk.model.DescribeEnvironmentsRequest;

@Path("/admin/leader")
public class LeaderResource extends BaseResource {
    @Inject
    AmazonEC2 amazonEC2;

    @Inject
    AWSElasticBeanstalk elasticBeanstalk;

    @GET
    public String getLeader() throws Exception {
        /*
         * Avoid running if we're not in AWS after all
         */
        try {
            IOUtils.toString(new URL(
                    "http://169.254.169.254/latest/meta-data/instance-id")
                    .openStream());
        } catch (Exception exc) {
            return "i-FFFFFFFF/localhost";
        }

        String environmentName = getMyEnvironmentName();

        List<Instance> environmentInstances = getInstances(
                "tag:elasticbeanstalk:environment-name", environmentName,
                "tag:leader", "true");

        if (environmentInstances.isEmpty()) {
            environmentInstances = getInstances(
                    "tag:elasticbeanstalk:environment-name", environmentName);

            Collections.shuffle(environmentInstances);

            if (environmentInstances.size() > 1)
                environmentInstances.removeAll(environmentInstances.subList(1,
                        environmentInstances.size()));

            amazonEC2.createTags(new CreateTagsRequest().withResources(
                    environmentInstances.get(0).getInstanceId()).withTags(
                    new Tag("leader", "true")));
        } else if (environmentInstances.size() > 1) {
            DeleteTagsRequest deleteTagsRequest = new DeleteTagsRequest().withTags(new Tag().withKey("leader").withValue("true"));

            for (Instance i : environmentInstances.subList(1,
                        environmentInstances.size())) {
                deleteTagsRequest.getResources().add(i.getInstanceId());
            }

            amazonEC2.deleteTags(deleteTagsRequest);
        }

        return environmentInstances.get(0).getInstanceId() + "/" + environmentInstances.get(0).getPublicIpAddress();
    }

    @GET
    @Produces("text/plain")
    @Path("am-i-a-leader")
    public boolean isLeader() {
        /*
         * Avoid running if we're not in AWS after all
         */
        String myInstanceId = null;
        String environmentName = null;
        try {
            myInstanceId = IOUtils.toString(new URL(
                    "http://169.254.169.254/latest/meta-data/instance-id")
                    .openStream());

            environmentName = getMyEnvironmentName();
        } catch (Exception exc) {
            return false;
        }

        List<Instance> environmentInstances = getInstances(
                "tag:elasticbeanstalk:environment-name", environmentName,
                "tag:leader", "true", "instance-id", myInstanceId);

        return (1 == environmentInstances.size());
    }

    protected String getMyEnvironmentHost(String environmentName) {
        return elasticBeanstalk
                .describeEnvironments(
                        new DescribeEnvironmentsRequest()
                                .withEnvironmentNames(environmentName))
                .getEnvironments().get(0).getCNAME();
    }

    private String getMyEnvironmentName() throws IOException,
            MalformedURLException {
        String instanceId = IOUtils.toString(new URL(
                "http://169.254.169.254/latest/meta-data/instance-id"));

        /*
         * Grab the current environment name
         */
        DescribeInstancesRequest request = new DescribeInstancesRequest()
                .withInstanceIds(instanceId)
                .withFilters(
                        new Filter("instance-state-name").withValues("running"));

        for (Reservation r : amazonEC2.describeInstances(request)
                .getReservations()) {
            for (Instance i : r.getInstances()) {
                for (Tag t : i.getTags()) {
                    if ("elasticbeanstalk:environment-name".equals(t.getKey())) {
                        return t.getValue();
                    }
                }
            }
        }

        return null;
    }

    public List<Instance> getInstances(String... args) {
        Collection<Filter> filters = new ArrayList<Filter>();

        filters.add(new Filter("instance-state-name").withValues("running"));

        for (int i = 0; i < args.length; i += 2) {
            String key = args[i];
            String value = args[1 + i];

            filters.add(new Filter(key).withValues(value));
        }

        DescribeInstancesRequest req = new DescribeInstancesRequest()
                .withFilters(filters);
        List<Instance> result = new ArrayList<Instance>();

        for (Reservation r : amazonEC2.describeInstances(req).getReservations())
            result.addAll(r.getInstances());

        return result;
    }
}

score 1 · Answer 2 · answered Feb 22 '14 at 16:35

1

You can keep a secret URL (a long URL is un-guessable, almost as safe as a password), hit this URL from somewhere. On this you can execute the task.

One problem however is that if the task takes too long, then during that time your server capacity will be limited. Another approach would be for the URL hit to post a message to the AWS SQS. The another EC2 can have a code which waits on SQS and execute the task. You can also look into http://aws.amazon.com/swf/

answered Feb 22 '14 at 16:35

Sony Kadavan

3,982
2
19
26

1

Long URL is not (almost) as safe as a password, URL hits are stored in the access log, while password does not. – Yonatan Sep 29 '14 at 07:51

score 0 · Answer 3 · answered May 22 '14 at 00:44

Another approach if you're running on the Linux-type EC2 instance:

Write a shell script that does (or triggers) your periodic task
Leveraging the .ebextensions feature to customize your Elastic Beanstalk instance, create a container command that specifies the parameter leader_only: true -- this command will only run on an instance that is designated the leader in your Auto Scaling group
Have your container command copy your shell script into /etc/cron.hourly (or daily or whatever).

The result will be that your "leader" EC2 instance will have a cron job running hourly (or daily or whatever) to do your periodic task and the other instances in your Auto Scaling group will not.

When your leader is killed then you will have no cron jobs running, because the secondaries will not automatically pick it up. — erjiang, Jan 09 '15 at 19:03
@erjiang: is this still true? [This description from the docs](https://stackoverflow.com/a/38880099) suggests the other instances *will* automatically pick up. — djvg, Oct 25 '18 at 06:51

With Elastic Beanstalk, can I determine programmatically if I'm on the leader node?

3 Answers3