0

I use a server with 16 cores, 64 GB ram, 2.5 TB disk and I want to execute a Giraph program. I have installed hadoop-2.7.2 and I don't know how can configure hadoop to use only a partial amount of server resources because the server used by many users. Requirements: Hadoop must use max 12 cores (=> 4 cores for NameNode, DataNode, JobTracker, TaskTracker and max 8 for tasks) and max 28GB ram (i.e., 4*3GB + 8*2GB).

My Yarn-site resources configuration:

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>28672</value>
<description>Physical memory, in MB, to be made available to running containers</description>
</property>

<property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>12</value>
    <description>Number of CPU cores that can be allocated for containers.</description>
</property>
</configuration>

When I try to execute Giraph program, in http://localhost:8088 Yarn Application state is: ACCEPTED: waiting for AM container to be allocated, launched and register with RM.

I think some configurations are missing from my Yarn-site.xml in order to adapt the above requirements.

user349
  • 11
  • 4

1 Answers1

0

Before assigning resources to the services take a look at Yarn tuning Guide file from Cloudera, you will get idea how much resources should be allocated to OS, Hadoop daemons, etc

As you mentioned

Yarn Application state is: ACCEPTED: waiting for AM container to be allocated, launched and register with RM

If there is no available resources for a job, then it will be in ACCEPTED state until it get resources. So in your case, check how many jobs are submitting at the same time and check the resources utilisation for those jobs.

If you want to configure no waiting for your jobs, you have to consider creating scheduler queues

BruceWayne
  • 3,286
  • 4
  • 25
  • 35