0

I'm trying to run a container on a RHEL 6.5 but I keep hitting this problem:

sudo docker run -u postgres -it registry/postgres /bin/bash
/bin/bash: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: Permission denied

When run as user 'root', the container starts fine but the problem appears again when trying to switch to another user:

$ sudo docker run -u root -it registry/database /bin/bash
[root@8a20410eaa5e /]# su postgres
su: /bin/bash: Permission denied

This is a specific container built by us, based on CentOS 6.5 an that runs Postgres. The Dockerfile to build it has "USER postgres" in it, and it works fine elsewhere except these servers. I can reproduce the same behaviour with a busybox container:

$ sudo docker run -u nobody -it 10.188.13.136:8080/busybox
/ $ ls
/bin/sh: ls: Permission denied

The RHEL 6.5 host has SELinux enabled. We have other other hosts where SELinux and this container works fine there. The audit log for this host looks clean, no error messages that I can see when trying to run the container.

This is what we've tried so far:

  • update the SELinux policies in RHEL ("sudo yum upgrade selinux-policy"), as they were not the latest versions
  • get SELinux into permissive mode (setenforce 0); not tried to switch it off completely and reboot
  • start the Docker daemon with "--selinux-enabled=true"
  • start the container with --privileged
  • start the container with --security-opt=:label:disable
  • we're running the latest RHEL 6.5 kernel: 2.6.32-504.16.2.el6.x86_64

Also run a strace session for the 'su ' command within the container but could not see much beyond these:

 17    setgid(10000)                     = 0
 17    setuid(10000)                     = 0
 17    munmap(0x7f07a3540000, 2101304)   = 0
 17    munmap(0x7f07a311c000, 2113776)   = 0
 17    munmap(0x7f07a2f03000, 2196352)   = 0
 17    munmap(0x7f07a2cea000, 2198192)   = 0
 17    munmap(0x7f07a2ae8000, 2101272)   = 0
 17    munmap(0x7f07a28e4000, 2109624)   = 0
 17    munmap(0x7f07a26e0000, 2109672)   = 0
 17    munmap(0x7f07a24d3000, 2148896)   = 0
 17    munmap(0x7f07a22d0000, 2105488)   = 0
 17    munmap(0x7f07a20cb000, 2113848)   = 0
 17    munmap(0x7f07a1ec5000, 2118168)   = 0
 17    munmap(0x7f07a3321000, 2221912)   = 0
 17    execve("/bin/bash", ["bash"], [/* 15 vars */]) = -1 EACCES (Permission denied)
 17    write(2, "su: ", 4)               = 4
 17    write(2, "/bin/bash", 9)          = 9

The full strace dump is here in case it's needed: http://pastebin.com/42C2B8LP.

We're not sure what to look for next, any ideas?

Oscar Renalias
  • 1
  • 2
  • 1
    have you tried to check you auditd logs? /var/log/audit/audit.log, put selinux in permissive mode, in this way you can see if the problem is selinux. – c4f4t0r May 15 '15 at 09:56
  • We did, both. Setting SElinux to permissive (setenforce 0) and then running the container did not seem to help. I cannot see anything unusual in the audit logs either when running the container: http://pastebin.com/F6LSUn2J (this is from an unsuccessful run with "sudo docker run -u postgres -it registry/postgres /bin/bash", which returns "/bin/bash: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: Permission denied") – Oscar Renalias May 15 '15 at 10:38
  • I also forgot to state that this is happening on Docker 1.6 from EPEL-testing. We did downgrade to 1.3.2 to verify if it was an issue with Docker but the behaviour was still reproducible. – Oscar Renalias May 15 '15 at 11:55
  • How did you set up your storage? – Michael Hampton May 15 '15 at 15:09
  • @MichaelHampton it's under /var/lib/docker, default location. /var is an LVM volume group: `/dev/mapper/vg_main-lv_var on /var type ext4 (rw)`. It's the same in every host. Or did you mean something else? – Oscar Renalias May 15 '15 at 21:36
  • Also tried to disable SELinux and rebooted the server (getenforce reports `Disabled`), but that made no difference and the issue is still present. – Oscar Renalias May 19 '15 at 12:00
  • You really should deploy this on RHEL 7. I can't think of any good reason to do this on RHEL 6. – Michael Hampton May 25 '15 at 13:06
  • Indeed. But RHEL 6.5 is what our client is currently running and there's no option at the moment to upgrade. The strangest thing is that it's worked perfectly well so far for tens of servers; only the latest ones have experienced these weird issues. – Oscar Renalias May 26 '15 at 07:17

1 Answers1

0

I was finally able to solve this problem. Which means that it seems that I found a solution, but I'm not still sure what the problem is:

1) pull container from registry 2.0 + run with docker 1.6 -> fail

2) pull container from registry 0.9.x (either Docker's own or the Artifactory server that we run internally) + run with docker 1.6 -> works

3) pull container from registry 2.0 + run with docker 1.5 or older -> fail

4) pull container form registry 0.9.x + run with docker 1.5 or newer -> works

I don't really think that Registry 2.0 is at fault but I have no better answer. The new registry is much faster than the old one but guess we'll stick with the older registry for the time being.

Oscar Renalias
  • 1
  • 2
  • Did you find out the root cause? The same thing happened to me RHEL 6.8, docker 1.7.1. How did you force docker to pull from v1? – Nam Ngo Apr 07 '16 at 01:04
  • No, we did not find the root cause. The way we "forced" a v1 registry was to run the old registry legacy image and not the newer v2 one (use https://hub.docker.com/_/registry/ and use a label lower than "2"), and it's been fine since then. – Oscar Renalias Apr 10 '16 at 18:56