2

Requirement: Restore DynamoDB table from S3 Backup location.

We created Data Pipeline job, and then edit Resources section in Architect Wizard.

We placed 20 instances under Core Instance count, but after the Data Pipeline job activation, EMR Cluster was created with only one master and one core instance.

Could you please suggest us, how to increase the number of cores under Resources section

u1234
  • 101
  • 1
  • 12

1 Answers1

0

You might be hitting the total EC2 resource limit. You can have only 20 On-Demand EC2 instances running unless you have requested a limit raise. Datapipeline acknowledges this limit. You have to increase the EC2 Ondemand resource limit using the link below.

A similar scenario from the docs:

If you configure AWS Data Pipeline to automatically create a 20-node Amazon EMR cluster to process data and your AWS account has an EC2 instance limit set to 20, you may inadvertently exhaust your available backfill resources. As a result, consider these resource restrictions in your design or increase your account limits accordingly.

If you require additional capacity, you can use the Amazon Web Services Support Center request form to increase your capacity.

franklinsijo
  • 17,784
  • 4
  • 45
  • 63
  • We don't have any other active EC2 instances. Is it a right way of increasing Core instances count? or do we have other way to edit and increase the core instance count. – u1234 Feb 27 '17 at 05:34
  • When creating an EMR resource in Datapipeline, passing the number under CoreInstanceCount is the right way to request core instances. In addition to the master instance count, you are requesting for a total of 21 instances. What is the instance type requested for master and core? Were there any task instances? – franklinsijo Feb 27 '17 at 05:53
  • There will be no option to specify master instance count right? Correct me If I am wrong. Also I did not see task instances specification as well. – u1234 Feb 27 '17 at 10:06
  • You cannot specify master instance count, that is 1 by default and cannot be modified. But there are Task Instance Type and Task Instance count options available. Since you did not add them, you happen to have requested 21 instances in total. And what is the type of instance requested? – franklinsijo Feb 27 '17 at 10:11
  • I have specified Task Instance Count: 5 (They are provisioning 5 instances- just checked now). The problem is with Core instances count, its not changing from 1. We have used "m3.xlarge" type for both Task instance and Core Instance. Will there be any performance increase by specifying Task Instance count >1 (They are provisioning as we requested)? – u1234 Feb 27 '17 at 13:03
  • Well that is strange. If the instance can be provisioned for task nodes, it should work for core nodes also. It would be helpful if you contact AWS support on this! And with the node types, task nodes are meant only for processing (i.e.,) only nodemanager service runs whereas for core nodes both datanode and nodemanager runs. – franklinsijo Feb 27 '17 at 14:19