15

I'm facing a problem when using AWS ECS Fargate on a new AWS account. When I try to run run more than 5 tasks, I get an error saying:

service feedback was unable to place a task. Reason: You've reached the limit on the number of tasks you can run concurrently.

Screenshot of AWS ECS Console"

Alastair McCormack
  • 26,573
  • 8
  • 77
  • 100
ashraf revo
  • 767
  • 2
  • 12
  • 24
  • Were you able to figure this out? I get this at the deploys, even when I have set desired count to 1 and deployment `MaximumPercent` set to 100. I don't think this is related to ECS service limit cause I don't have any other ECS cluster or service. I suspect some config clashes going on inside that hasn't been documented. Meanwhile,I can't seem to pass through this, Cloudformation just gets stuck for hours to rollback (I use Cloudformation to deploy ECS). It's frustrating. – roxxypoxxy Nov 08 '18 at 15:04
  • This problem solved by email support team to increase the limit – ashraf revo Nov 08 '18 at 15:10
  • Did they actually say something about the limit? I mean you had actually crossed the limit of 50? – roxxypoxxy Nov 08 '18 at 15:12
  • No it only take for me 3 weaks to increase the limit – ashraf revo Nov 08 '18 at 15:15
  • Well my question was: you mentioned you were trying run a service with 5 tasks. Were you actually exceeding the specified limit of 50? How is it possible to exceed the limit with what you have, a service with 5 tasks (and as we can see desired count set to 1)? – roxxypoxxy Nov 08 '18 at 15:18
  • 50. No I only TRY with 10 tasks. You can set the limit as your need – ashraf revo Nov 08 '18 at 15:33

6 Answers6

15

TL;DR

Run an EC2 instance

More Detail

I raised a support request with AWS after receiving this on a new account. This is their response:

In this situation, in order to activate the requested regions i.e EU (Ireland) and/or other regions, please launch an EC2 instance in those regions, this will enable you to run multiple concurrent Fargate tasks.

Running an t3.medium instance for 10 minutes in eu-west-1 resulted in an email being sent to the account email informing us that:

Your request has now been validated for AWS Europe (Ireland) region(s).

After which, I was able to run more than 2 concurrent ECS Fargate tasks.

Don't forget to terminate your EC2 instance!

Even more detail: Run a basic EC2 instance:

  1. https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#LaunchInstances:
  2. Change to the same region as your ECS cluster
  3. Click Launch Instance
  4. Name: Test
  5. Image: Amazon Linux
  6. Architecture: 64-bit x86
  7. Instance Type: t3.medium
  8. Key pair: Proceed without
  9. Network: Uncheck all
  10. Storage: 8GB or default
  11. Click Launch
  12. Wait 10 minutes
  13. Goto https://eu-west-1.console.aws.amazon.com/ec2/v2/home?region=eu-west-1#Instances:
  14. Select correct region
  15. Instant State -> Terminate Instance -> Terminate
Alastair McCormack
  • 26,573
  • 8
  • 77
  • 100
  • Thanks! This solution worked for me. I feel that this is definitely a work-around to some internal configuration bug with AWS, but I'm glad it fixed my Fargate task limit issue. – Vincent Catalano Dec 02 '21 at 22:13
  • This helped us. But I can't wonder just... WTF! – skoll Jan 11 '22 at 13:08
  • 1
    This worked. I don't have a support account, but I got the email saying the region is validated anyway. You don't need to open a ticket or anything first, just run the ec2 instance for a bit and it'll work. – deontologician Feb 01 '22 at 18:30
  • I dont know how, but this approach clearly works – Nazar Ahmed Amjad Mar 22 '22 at 12:13
  • I think this approach is a little bit hacky. I don't know if in the future AWS would change this behavior or not but the more straightforward way would be the one answered by @jarmod – Turbo Jun 09 '22 at 05:34
  • @Turbo can you show us where in the Quotas and Limits pages where it specifies a limit of just 2x ECS Fargate tasks for a new account? Had it existed then I wouldn’t have needed to contact AWS support and they wouldn’t have given the steps shown above. – Alastair McCormack Jun 09 '22 at 06:59
  • @Alastair McCormack For Fargate, it has their own service limit separated from ECS service limit. Search for "Fargate" in service limit page and request for the new limit. Please checkout this link https://eu-central-1.console.aws.amazon.com/servicequotas/home/services/fargate/quotas Make sure to change the region before you request the raise. – Turbo Jun 09 '22 at 07:46
  • @Turbo where exactly does it show the current limit is 2? And what new limit do we request? – Alastair McCormack Jun 09 '22 at 07:48
  • @Alastair McCormack In my case it's not 2. It was 5. You can see the current limit in "Applied quota value" column For the new limit, depending on your requirement, you can request up to 1,000. In my case, I just requested for the maximum. – Turbo Jun 09 '22 at 07:49
  • BTW, for the request, this can be done from the console. There is a button request quota increase when you click inside the service you want to raise the limit. This video might help clarify what I meant. https://www.youtube.com/watch?v=Ddf5WG5I0Qo – Turbo Jun 09 '22 at 07:55
  • @Turbo it sounds like AWS has now exposed this "new account" limit in the console where previously it was hidden (I'd be surprised if this limit is documented anywhere else). I'll experiment with a new account and report back. – Alastair McCormack Jun 09 '22 at 09:00
11

Apparently all new account has a limit of 2 tasks that can be run concurrently, which does not seem to be documented anywhere. Increasing the limit fixes the issue.

roxxypoxxy
  • 2,973
  • 1
  • 21
  • 28
10

All AWS accounts have service limits, to protect you and other accounts.

See the ECS Service Limits page and decide if you need to request an increase in service limits.

In your case, you're using Fargate which has their own service limit separated from ECS service limit. Search for "Fargate" in service limit page and request for the new limit.

enter image description here

Turbo
  • 548
  • 5
  • 13
jarmod
  • 71,565
  • 16
  • 115
  • 122
  • Isn't the default rate 50 though? By default, he should be able to run more than 5 tasks – johnnyodonnell Nov 05 '18 at 20:54
  • 1
    @johnnyodonnell The current limits are 50 and, if I recall, they were 20 before that. I'd guess that he may be running other workloads that in aggregate caused him to reach the per region, per account soft limit (whatever it was at the time). It's not impossible that AWS service limit checking was bugged, but the OP's gone quiet and I've seen no evidence of it. – jarmod Nov 05 '18 at 21:46
  • I get this issue over and over again. I get this at the deploys, even when I have set desired count to 1. I don't think this is related to ECS service limit cause I don't have any other ECS cluster or service. I suspect some config clashes going on inside that hasn't been documented. Meanwhile,I can't seem to pass through this, Cloudformation just gets stuck for hours to rollback (I use Cloudformation to deploy ECS). It's very frustrating. – roxxypoxxy Nov 08 '18 at 15:05
  • Seems like we are not the only ones https://forums.aws.amazon.com/thread.jspa?messageID=875584󕱀 https://www.reddit.com/r/aws/comments/8b3thk/im_only_limited_to_two_ecs_fargate_tasks_on_any/ – roxxypoxxy Nov 08 '18 at 15:10
  • I talked with the support team, apparently all new account has a limit of 2 tasks that can be run concurrently, which does not seem to be documented anywhere. Increasing the limit fixed my issue. – roxxypoxxy Nov 15 '18 at 09:50
  • 1
    @roxxypoxxy is right, the default limit is very, very low and is not documented. – ritmatter Nov 24 '21 at 15:10
  • 2
    What was the name of the quota you increased? In ECS or Fargate? – twinlakes Jan 27 '22 at 04:38
0

From the documentation:

New AWS accounts might start with limits that are lower than the limits described here. These limits can increase over time.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-limits.html

MarZab
  • 2,543
  • 21
  • 28
0

This question led me to solve a related problem that I was facing, so I wanted to point it out here. If someone in the future end up here with the same problem that I had, it can help.

If you are trying to deploy more than one service stack that runs tasks definitions on ECS for a application running in a docker image, and you are facing this:

error log image

The problem described here in this question may be the reason. You can check it by going to the ECS console, and searching for the service that is failing, it will show something related to "you've reached the maximum number of concurrently tasks...".

EDIT - PROBLEM SOLVED

I solved this problem by contacting the AWS support team. You just have to contact them by searching in the services list for:

AWS support -> Create Case -> Select that your problem is related to the ECS and related to Service quotas.

I had to open a case saying that I wanted to increase the number of quotas, wich by default is 5.000 (2022) but as we can see, if you are facing this error it isn't for some accounts.

So I opened the case requesting for 5.001 quotas, and them I explained the problem that I was facing. They said that it isn't a common error, but it happens with some accounts.

Pedro Luiz
  • 321
  • 1
  • 5
-1

Running an EC2 instance for some amount of time seemed to have increased the configuration for ECS Fargate tasks.

Tae Min Ha
  • 1
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community May 10 '22 at 11:04
  • How is this answer different to my answer saying the same thing? – Alastair McCormack Jun 09 '22 at 07:03