How to delete a queue in YARN?

Question

I see a queue in the yarn configuration file that I want to delete:

<property>
  <name>yarn.scheduler.capacity.root.queues</name>
  <value>a,b,c</value>
  <description>The queues at the this level (root is the root queue).
  </description>
</property>

<property>
  <name>yarn.scheduler.capacity.root.a.queues</name>
  <value>a1,a2</value>
  <description>The queues at the this level (root is the root queue).
  </description>
</property>

<property>
  <name>yarn.scheduler.capacity.root.b.queues</name>
  <value>b1,b2,b3</value>
  <description>The queues at the this level (root is the root queue).
  </description>
</property>

Say I want to remove queue c. I remove c from the list under the line <name>yarn.scheduler.capacity.root.queues</name> so that line looks like this:

<value>a,b,c</value>

Then I go to the command line and run yarn rmadmin -refreshQueues.

But I get the following error message:

Caused by java.io.IOException: c cannot be found during refresh!`

I'm trying to delete the queue c. How do I delete it?

I noticed here it says that

Note: Queues cannot be deleted, only addition of new queues is supported - the updated queue configuration should be a valid one i.e. queue-capacity at each level should be equal to 100%.

...so, how do I delete a queue if I don't need it anymore?

thanks.

From that page: "Administrators can add additional queues at runtime, but queues cannot be deleted at runtime." It sounds like you cannot delete at runtime, BUT you can stop/kill the YARN, update the config file (and remove `c`) and run YARN with the new configuration. — Arash, Mar 06 '17 at 07:55
That is a known issue: https://issues.apache.org/jira/browse/YARN-4022 — Geethu Jose, Mar 06 '17 at 08:42

Arash · Answer 1 · 2017-03-06T20:21:49.060

From that page:

Administrators can add additional queues at runtime, but queues cannot be deleted at runtime.

It sounds like you cannot delete at runtime, BUT you can stop/kill the YARN, update the config file (and remove c) and run YARN with the new configuration. So if you can afford to stop/start the YARN then here is the process:

Note: Before killing the YARN Read here on the effects on the system depending on your YARN version.

Here goes the process:

Stop the MapReduce JobHistory service, ResourceManager service, and NodeManager on all nodes where they are running, as follows:

 sudo service hadoop-mapreduce-historyserver stop
 sudo service hadoop-yarn-resourcemanager stop
 sudo service hadoop-yarn-nodemanager stop

Then Edit the config file and remove c

Start the MapReduce JobHistory server, ResourceManager, and NodeManager on all nodes where they were previously running, as follows:

 sudo service hadoop-mapreduce-historyserver start
 sudo service hadoop-yarn-resourcemanager start
 sudo service hadoop-yarn-nodemanager start

Commands are from here

@Sother for making other changes (like `add`) but not for `delete`. — Arash, Mar 10 '17 at 22:12

score 1 · Answer 2 · answered Mar 06 '17 at 11:15

1

You have to stop Yarn if you want to delete a queue. Only add, update and stop is supported while Yarn is running.

answered Mar 06 '17 at 11:15

YunjieJi

45
2
11

how do I stop yarn? – makansij Mar 06 '17 at 17:00
and if I stop it, will it kill all of my applications running? – makansij Mar 06 '17 at 17:18
No, Yarn won't immediately kill you apps. Once you stop a queue, no more app can be added to it, and yarn will wait for all apps on this queue completed. You can see the [official tutorial](http://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html) for more details. – YunjieJi Mar 07 '17 at 01:36
That isn't true. I just tried it. YARN does actually kill your applications if you restart `hadoop-yarn-nodemanager` while the application is running – makansij Apr 04 '17 at 22:58

How to delete a queue in YARN?

2 Answers2