4

I have a nodeBB install which relies on Redis as the data store. Its just a test install at the moment while I iron out any problems. The Redis instance stayed up for a few days but then fell over with the following errors in the logs:

3693:C 01 Dec 03:34:22.056 # Error moving temp DB file on the final destination: Operation not permitted
7089:M 01 Dec 03:34:22.155 # Background saving error
7089:M 01 Dec 03:34:28.067 * 1 changes in 900 seconds. Saving...
7089:M 01 Dec 03:34:28.068 * Background saving started by pid 3699
3699:C 01 Dec 03:34:28.069 # Error moving temp DB file on the final destination: Operation not permitted
7089:M 01 Dec 03:34:28.168 # Background saving error
7089:M 01 Dec 03:34:34.080 * 1 changes in 900 seconds. Saving...
7089:M 01 Dec 03:34:34.081 * Background saving started by pid 3700
3700:C 01 Dec 03:34:34.083 # Error moving temp DB file on the final destination: Operation not permitted
7089:M 01 Dec 03:34:34.181 # Background saving error

Redis was installed as root on Centos 6.7 using the following instructions (the default yum package is too old) using the following method:

tar xzf redis-3.0.x.tar.gz
cd redis-3.0.1
make
make test
make install
cd utils
chmod +x install_server.sh
./install_server.sh

Working directory as set in /etc/redis/6379.conf :

# The filename where to dump the DB
dbfilename dump.rdb

# The working directory.
#
# The DB will be written inside this directory, with the filename specified
# above using the 'dbfilename' configuration directive.
#
# The Append Only File will also be created inside this directory.
#
# Note that you must specify a directory here, not a file name.
dir /var/lib/redis/6379

Can anyone suggest what would be causing this and how best to resolve the issue? Where will Redis try and save temp files, which user would it use and which permissions would safely resolve this?

Also of interest would be a free method of monitoring the Redis instance so I know if / when it falls over.

6359.conf:

daemonize yes

pidfile /var/run/redis_6379.pid

port 6379

tcp-backlog 511

timeout 0

tcp-keepalive 0

loglevel notice

logfile /var/log/redis_6379.log


databases 16

save 900 1
save 300 10
save 60 10000

stop-writes-on-bgsave-error yes

rdbcompression yes

rdbchecksum yes

dbfilename dump.rdb

dir /var/lib/redis/6379


slave-serve-stale-data yes

slave-read-only yes

repl-diskless-sync no

repl-diskless-sync-delay 5

repl-disable-tcp-nodelay no

slave-priority 100


appendonly no


appendfilename "appendonly.aof"


appendfsync everysec


no-appendfsync-on-rewrite no


auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

aof-load-truncated yes


lua-time-limit 5000


slowlog-log-slower-than 10000

slowlog-max-len 128

latency-monitor-threshold 0

notify-keyspace-events ""

hash-max-ziplist-entries 512
hash-max-ziplist-value 64

list-max-ziplist-entries 512
list-max-ziplist-value 64

set-max-intset-entries 512

zset-max-ziplist-entries 128
zset-max-ziplist-value 64

hll-sparse-max-bytes 3000

activerehashing yes

client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

hz 10
codecowboy
  • 1,307
  • 7
  • 18
  • 31
  • What's in the audit log? – Michael Hampton Dec 01 '15 at 09:14
  • where would I find audit.log? Its not in /var/log – codecowboy Dec 01 '15 at 09:17
  • @codecowboy, Can you check your directory and file permission where the redis db is supposed to be? `ls -al /var/lib/redis/6379`. Try setting `/var/lib/redis/6379` dir ownership to redis user/group, dir permission `750` and db file permission 644. Also make sure you have free disk space. – Diamond Dec 04 '15 at 13:09
  • @bangal thanks. I have done that and will let you know if it helps – codecowboy Dec 06 '15 at 10:01
  • @bengal. This did not resolve the problem. I am still getting Error moving temp DB file on the final destination: Operation not permitted in the redis logs. – codecowboy Dec 08 '15 at 09:54
  • @codecowboy, well, the error is clearly permission related, but there can be other issues too. I guess, you should post more info on the db directory and your full redis.conf file to get further help. – Diamond Dec 08 '15 at 12:31

3 Answers3

0

Check if you have SELinux enabled with sestatus. If it is enabled (something that is recommended) please put it in permissive mode via setenforce 0.

After that check again your system and I guess that the error will disappear, check the security log to see what was blocked by SELinux, enable it in your policy and put SELinux back in enforcing mode (of course if you want to have your server secured by SELinux)

scai
  • 2,269
  • 2
  • 13
  • 16
Rodrigo Gonzalez
  • 1
  • SELinux is disabled at the moment. Somebody else answered this before the bounty started I think but then deleted their answer. – codecowboy Dec 09 '15 at 07:22
  • OK, 2 ideas I have in the order I would apply them. 1 - change log level to debug in your configuration and see if you get more information about the issue. 2 - if everything else fails, strace to the server pid and check or put here the output before the error to be able to know the system call that is failing. – Rodrigo Gonzalez Dec 09 '15 at 21:01
0

The OP is unclear whether the background save ever succeeded before the errors began, but I've seen this error associated with disk space issues - at the chance of stating the obvious, check with df if you've ran out of space.

IMPORTANT - the conf you've attached suggests that your server is not using password authentication (the requirepass directive), does not rename the admin commands and isn't bound to a specific interface (the bind directive). This is a potential security risk - more at http://antirez.com/news/96.

Itamar Haber
  • 894
  • 8
  • 8
0

Redis is very data backup friendly since you can copy RDB files while the database is running: the RDB is never modified once produced, and while it gets produced it uses a temporary name and is renamed into its final destination atomically using rename(2) only when the new snapshot is complete. I think redis or system is not healthy above process doesn't required any additional privileges until redis is not configured properly or system issues.

asktyagi
  • 2,860
  • 2
  • 8
  • 25