starting mongodb via numactl as daemon

Question

I'm trying to get mongodb started on a NUMA machine as a daemon. When I run

numactl --interleave=all mongod &

Mongo starts and runs correctly, but all the output still shows up. (e.g., Fri Jun 22 12:10:29 [initandlisten] connection accepted from 127.0.1.1:51837)

However, when I start mongo on its own (like below), it fails (logs below):

service mongodb start

I get the following in the logs

Fri Jun 22 12:08:41 [initandlisten] MongoDB starting : pid=3348 port=27017 dbpath=/var/lib/mongodb 64-bit host=beckett
Fri Jun 22 12:08:41 [initandlisten]
Fri Jun 22 12:08:41 [initandlisten] ** WARNING: You are running on a NUMA machine.
Fri Jun 22 12:08:41 [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
Fri Jun 22 12:08:41 [initandlisten] **              numactl --interleave=all mongod [other options]
Fri Jun 22 12:08:41 [initandlisten]
Fri Jun 22 12:08:41 [initandlisten] db version v2.0.6, pdfile version 4.5
Fri Jun 22 12:08:41 [initandlisten] git version: e1c0cbc25863f6356aa4e31375add7bb49fb05bc
Fri Jun 22 12:08:41 [initandlisten] build info: Linux ip-10-110-9-236 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Fri Jun 22 12:08:41 [initandlisten] options: { auth: "true", command: [ "run" ], config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
Fri Jun 22 12:08:41 [initandlisten] journal dir=/var/lib/mongodb/journal
Fri Jun 22 12:08:41 [initandlisten] recover : no journal files present, no recovery needed
Fri Jun 22 12:08:42 [initandlisten] couldn't open /var/lib/mongodb/admin.ns errno:13 Permission denied
Fri Jun 22 12:08:42 [initandlisten] error couldn't open file /var/lib/mongodb/admin.ns terminating
Fri Jun 22 12:08:42 dbexit:
Fri Jun 22 12:08:42 [initandlisten] shutdown: going to close listening sockets...
Fri Jun 22 12:08:42 [initandlisten] shutdown: going to flush diaglog...
Fri Jun 22 12:08:42 [initandlisten] shutdown: going to close sockets...
Fri Jun 22 12:08:42 [initandlisten] shutdown: waiting for fs preallocator...
Fri Jun 22 12:08:42 [initandlisten] shutdown: lock for final commit...
Fri Jun 22 12:08:42 [initandlisten] shutdown: final commit...
Fri Jun 22 12:08:42 [initandlisten] shutdown: closing all files...
Fri Jun 22 12:08:42 [initandlisten] closeAllFiles() finished
Fri Jun 22 12:08:42 [initandlisten] journalCleanup...
Fri Jun 22 12:08:42 [initandlisten] removeJournalFiles
Fri Jun 22 12:08:42 [initandlisten] shutdown: removing fs lock...
Fri Jun 22 12:08:42 dbexit: really exiting now

I don't know how admin.ns could have a permissions issue while I'm running as root or why when wrapped in numactl it starts up ok. Ideally, I'd like to use numactl in the start_server() function like so:

start_server(){
    /usr/bin/numactl --interleave=all -- \
    start-stop-daemon --background --start --quiet --pidfile $PIDFILE \
        --make-pidfile --chuid $DAEMONUSER \
        --exec $DAEMON -- $DAEMON_OPTS
    errcode=$?
    return $errcode
}

Bottom line, how can I get mongo to start as a daemon on a NUMA machine?

See also http://unix.stackexchange.com/questions/187474/ , which takes into account some changes to the MongoDB package in more recent years. — JdeBP, Mar 02 '15 at 12:01

score 5 · Accepted Answer · answered Jun 26 '12 at 19:06

5

Turns out my problem was a combination of numa and permissions issues. Thanks, @Mark for the help. To start mongodb as a daemon on a NUMA setup, replace the start_server() function in /etc/init.d/mongodb with the following:

start_server() {
# check for numactl
NUMACTL=$(which numactl)
if [ -n "$NUMACTL" ]; then
    DAEMON_OPTS="--interleave=all ${DAEMON} ${DAEMON_OPTS}"
    DAEMON="$NUMACTL"
fi

# Start the process using the wrapper
            /usr/bin/numactl --interleave=all -- \
            start-stop-daemon --background --start --quiet --pidfile $PIDFILE \
                        --make-pidfile --chuid $DAEMONUSER \
                        --exec $DAEMON -- $DAEMON_OPTS
            errcode=$?
        return $errcode
}

answered Jun 26 '12 at 19:06

Libby

581
2
6
21

I'm not running a NUMA system AFAIK so this may be off-topic for the OP, but in case it's helpful to anyone...on Ubuntu at least, the ownership of all the files in MongoDB's data directory (on Ubuntu the default data directory is /var/lib/mongodb) should be mongodb:nogroup. Somehow mine got set to root:root (probably because I ran mongod manually to test something); changing the ownership made `sudo start mongodb` (and `sudo service start mongodb`) work correctly again. – Matt Browne Feb 28 '14 at 03:04
So, this was the command that fixed it for me: `sudo chown -R mongodb:nogroup /var/lib/mongodb/*`. Note that /var/lib/mongodb is Ubuntu's default setting and may not be the setting for other distros; MongoDB's own default data dir is /data/db. – Matt Browne Feb 28 '14 at 03:07

score 3 · Answer 2 · answered Jun 25 '12 at 12:54

I presume you know the usual warnings (http://www.mongodb.org/display/DOCS/NUMA) about 'mongo & numa' so I won't go on about them.

Here's a sample upstart configuration file for mongodb with numa - https://gist.github.com/1364716.

Based on this Google Group thread, the following lines were added to the start_server function in the init script and it was successful -

start_server() { 
# check for numactl 
NUMACTL=$(which numactl) 
if [ -n "$NUMACTL" ]; then 
    DAEMON_OPTS="--interleave=all ${DAEMON} ${DAEMON_OPTS}" 
    DAEMON="$NUMACTL" 
fi 

# Start the process using the wrapper

That's the Group where I found the start_service function in my post, but I get the "couldn't open /var/lib/mongodb/admin.ns errno:13 Permission denied" error when I try "service mongodb start" with the "check for numactl" and "start the process using the wrapper" code bits included. — Libby, Jun 26 '12 at 18:40

score 1 · Answer 3 · answered Oct 28 '16 at 12:07

for ubuntu 16.04

[Unit]
Description=High-performance, schema-free document-oriented database
After=time-sync.target network.target

[Service]
Type=forking
User=mongod
Group=mongod
LimitNOFILE=65000
PermissionsStartOnly=true
EnvironmentFile=/etc/default/mongod
ExecStartPre=/usr/bin/percona-server-mongodb-helper.sh
ExecStart=/usr/bin/env bash -c "numactl --interleave=all /usr/bin/mongod $OPTIONS > ${STDOUT} 2> ${STDERR}"
#ExecStart=/usr/bin/env bash -c "/usr/bin/mongod $OPTIONS > ${STDOUT} 2> ${STDERR}"
PIDFile=/var/run/mongod.pid

[Install]
WantedBy=multi-user.target

score 0 · Answer 4 · answered Nov 17 '15 at 14:35

the two solutions above did no work for me, so here follows what got my mongodb running:

replace the start_server() function in /etc/init.d/mongodb for the code below

start_server() {
    test -e "$RUNDIR" || install -m 755 -o mongodb -g mongodb -d "$RUNDIR"

    NUMACTL=$(which numactl)

    if [ ! "$NUMACTL" ]; then
        # start original
        start-stop-daemon --background --start --quiet --pidfile $PIDFILE --make-pidfile --chuid $DAEMONUSER --exec $DAEMON -- $DAEMON_OPTS
        errcode=$?
        return $errcode
    else
        # Start the process using the wrapper
        $NUMACTL --interleave=all -- start-stop-daemon --background --start --quiet --pidfile $PIDFILE --make-pidfile --chuid $DAEMONUSER --exec $DAEMON -- $DAEMON_OPTS
        errcode=$?
        return $errcode
    fi
}

score 0 · Answer 5 · answered Aug 04 '16 at 07:48

0

for ubuntu 14

# apt-get -y install numactl

add to /etc/sysctl.conf line vm.zone_reclaim_mode = 0

# sysctl -p

# service mongod restart

answered Aug 04 '16 at 07:48

Hett

3,484
2
34
51

starting mongodb via numactl as daemon

5 Answers5