5

We are encountering a bizarre issue were non-root users are unable to execute any files (scripts or binaries) in certain directories. This transcript demonstrates the issue:

[root@b6 /]# mkdir q
[root@b6 /]# cp /bin/echo .
[root@b6 /]# cp /bin/echo q
[root@b6 /]# chown -R apps q
[root@b6 /]# ./echo ok
ok
[root@b6 /]# ./q/echo ok
ok
[root@b6 /]# su - apps
[apps@b6 ~]$ cd /
[apps@b6 /]$ ./echo ok
ok
[apps@b6 /]$ ./q/echo ok
-bash: ./q/echo: Permission denied
[apps@b6 /]$ ls -ld . echo q q/echo
dr-xr-xr-x. 29 root root  4096 Jun 10 00:34 .
-rwxr-xr-x.  1 root root 28176 Jun 10 00:34 echo
drwxr-xr-x.  2 apps root  4096 Jun 10 00:34 q
-rwxr-xr-x.  1 apps root 28176 Jun 10 00:34 q/echo
[apps@b6 /]# getfacl q q/echo
# file: q
# owner: apps
# group: root
user::rwx
group::r-x
other::r-x

# file: q/echo
# owner: apps
# group: root
user::rwx
group::r-x
other::r-x

[apps@b6 /]$ id
uid=2070(apps) gid=2070(apps) groups=2070(apps) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[apps@b6 /]$ uname -a
Linux b6.pdc 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug 28 17:19:38 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
[apps@b6 /]$ df / /q
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/md2       198166644 4273776 183826596   3% /
/dev/md2       198166644 4273776 183826596   3% /
[apps@b6 /]$ mount 
/dev/md2 on / type ext4 (rw)
...
[apps@b6 /]$ getenforce
Permissive
[apps@b6 /]$ strace q/echo ok
execve("q/echo", ["q/echo", "ok"], [/* 22 vars */]) = -1 EACCES (Permission denied)
dup(2)                                  = 3
fcntl(3, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f237b0a0000
lseek(3, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
write(3, "strace: exec: Permission denied\n", 32strace: exec: Permission denied
) = 32
close(3)                                = 0
munmap(0x7f237b0a0000, 4096)            = 0
exit_group(1)                           = ?

Our first thought was selinux, but this is off. Next was noexec on the f/s, but this isn't the case. Our user shell is just regular bash (not restricted).

AgileZebra
  • 151
  • 5
  • 1
    do you use acls? try: `getfacl q/echo` – Grizly Jun 10 '15 at 01:23
  • No acls being used. Will update the question... – AgileZebra Jun 10 '15 at 01:25
  • ok, next I would try `file /bin/echo` and make sure it matches q/echo, then try `strace q/echo ok` and see what it is doing, or failing to do. – Grizly Jun 10 '15 at 01:32
  • There is no difference between the 2 echo commands (it is freshly copied after all). file gives the same results for both; diff returns quietly. The same issue occurs if you create a script in the directory. Root can execute it, a user can't. The user can execute it however with "bash script" so it is readable but not executable in its own right. The executable bit is of course set and an absolute or relative path make no difference. Root can execute it fine without the explicit invocation via bash, so it's not a shebang issue. – AgileZebra Jun 10 '15 at 01:48
  • It doesn't seem to matter whether it is script or binary, or what the type of file is: it's as if execute permission is explicitly denied for all files inside the directory (or subdirectories) - as if it were a noexec f/s. I have updated the transcript with the strace output. – AgileZebra Jun 10 '15 at 01:48
  • A symbolic link to /bin/echo works; a hard link to it fails in the same way. – AgileZebra Jun 10 '15 at 01:55
  • Can't even execute strace.. useful. What about when logged in as another (non root) user, does it work then? Or is every non-root user failing to execute? [this suggest checking shared libraries](http://lkml.iu.edu/hypermail/linux/kernel/9810.2/0027.html) – Grizly Jun 10 '15 at 02:04
  • Found a similar issue: http://stackoverflow.com/a/20138875 I notice you haven't listed the mount info, which could be the key. IE, Possible duplicate of: http://stackoverflow.com/questions/20138711/weird-permission-denied-exception-when-running-bash-script-on-ec2-ubuntu – Grizly Jun 10 '15 at 02:12
  • No, the mount info is there already. It's just very short. It's on / . The file and directory that do not execute are on the same filesystem as /bin/ which do. It's all on / . All non-root users are similarly affected. – AgileZebra Jun 10 '15 at 03:32
  • I stand corrected. You did post it. I've honestly never used mount to view mounts before, I usually use /etc/fstab and /proc/mounts, `mount` uses /etc/mtab.. I'm guessing you've rebooted and this still occurs? – Grizly Jun 10 '15 at 04:45
  • Add `-n` to your `ls` outputs. I'm thinking some confused UID's maybe? EDIT: No, mode is +x for all so it can't be that. Does the same thing happen if you don't chown `/q/echo` to the `app` user? – fukawi2 Jun 10 '15 at 05:39
  • 1
    Does anything appear in log files (/var/log/secure, messages, others) when this fails? Could AppArmor be involved? I'm a bit vague about its capabilities but this feels like a restriction it could impose. – Paul Haldane Jun 10 '15 at 06:45
  • 1
    What version of Linux is this? – Paul Haldane Jun 10 '15 at 06:45
  • @paul, uname -a output is already in the transcript above. The distro is CentOS release 6.6 (Final). – AgileZebra Jun 10 '15 at 11:54
  • Aren't directories on partitions mounted with 'noexec' flag ? see in /etc/fstab. – tonioc Jun 10 '15 at 12:26
  • @tonioc, as per above, the directory in question is created on the root filesystem. The same filesystem were other binaries are executable. There is no noexec in effect. – AgileZebra Jun 10 '15 at 12:31

1 Answers1

0

I rechecked the logs (logging was misconfigured and previously there was no output when tailing messages - thanks Paul) and I've found this:

2015-06-10T12:25:15.547452+00:00 b6 kernel: [tpe] Denied untrusted exec of /q/echo (uid:2070) by /bin/bash (uid:2070), parents: /bin/su (uid:0), /bin/bash (uid:0), /usr/sbin/sshd (uid:0), /usr/sbin/sshd (uid:0), /sbin/init (deleted) (uid:0). Deny reason: directory uid not trusted

It appears we have grsecurity: Trusted Path Execution enabled (perhaps this is default in CentOS 6.6) and the owner of the directory is not trusted. Looks like we just need to add a number of users as trusted, or disable it as required.

AgileZebra
  • 151
  • 5