14

Searching for system-resolved service following the recent vulnerability disclosure, I came to see a very strange behavior from find command. 

 root@localhost:/# find . -name "*systemd-resolved*"
./usr/share/man/man8/systemd-resolved.service.8.gz
./usr/share/man/man8/systemd-resolved.8.gz

The command returns 0 or two lines as output for the first run. But if I run the command the second time I get:

root@localhost:/# find . -name "*systemd-resolved*"
./usr/share/man/man8/systemd-resolved.service.8.gz
./usr/share/man/man8/systemd-resolved.8.gz
./lib/systemd/systemd-resolved
./lib/systemd/system/systemd-resolved.service.d
./lib/systemd/system/systemd-resolved.service

This means the first time, "find" does not actually find everything. Also this only happens one time. Running the command next times shows correct output. I checked this on some other systems with Debian 8 (jessie) installed. On those with Kernel 4.9+ this exact problem always occurs but on systems with kernel 3.16 it doesn't happen.
After system reboot all this happen again. But the behavior is the same for each individual system. That means that if testing on a specific system returns (wrongly) two lines of output for first run and correct output for second run, then first run of the command after rebooting the system again prints 2 lines. So the systems show same behavior after each reboot (according to my tests). Files details are as follows:

-rw-r--r-- 1 root root  ./usr/share/man/man8/systemd-resolved.service.8.gz
lrwxrwxrwx 1 root root  ./usr/share/man/man8/systemd-resolved.8.gz -> systemd-resolved.service.8.gz
-rwxr-xr-x 1 root root  ./lib/systemd/systemd-resolved
drwxr-xr-x 2 root root  ./lib/systemd/system/systemd-resolved.service.d
-rw-r--r-- 1 root root  ./lib/systemd/system/systemd-resolved.service

EDIT: To all those who suggest the problem maybe related to this specific case for these specific files: "system-resolved" is just as example. This happens when searching other keywords too. This is another example which gives wrong results for the first run:

root@localhost:/# find . -name "*apache*"

Not anybody here is able to check this problem on a Debian 8 with latest kernel from backport repository?

user2808671
  • 183
  • 10
  • 2
    Can you try comparing traces of the two calls, for example using `strace`? Which OS did you observe the faulty behaviour on? What do you mean by "returns 0 or two results like above"? Zero or two lines of output, or exit code 0 + two lines? Does it happen again after starting a new shell or rebooting? It might be relevant that the first call only returns files, while the second returns files and directories. – l0b0 Jun 30 '17 at 13:38
  • 1
    @l0b0 As I said it happens on Debian with Kernel 4.9 in multiple systems. I did not check other distros. 0 or 2 means zero or two lines of output. It happens after every reboot. Your last statement does not apply here. It attempts to return everything. Both directories and files. – user2808671 Jun 30 '17 at 13:46
  • Sorry, what I meant was the system with kernel 3.16 also Debian? When you say the last statement does not apply, do you mean that it when it prints two paths it's two *arbitrary* paths out of the five that you expect, or is it always the same two? I don't understand what you mean by saying that it "attempts" to return everything. Can you also include the mount points for all the files? This is easy to find with `df /path1 /path2 …`. – l0b0 Jun 30 '17 at 13:52
  • 1
    @l0b0 Well I'm not sure what you're looking for, but as you can see I have mentioned the command so that one be able to reproduce the issue. That command must return all paths containing "systemd-resolved" but it won't. There are total five paths satisfying this condition but the "find" program only returns two of them or one or zero. What matters here is that the tool is giving wrong output and misses some correct paths. And as I mentioned I checked this on other systems with Debian, those with kernel 4.9 have this problem. This might be something serious beyond user space. – user2808671 Jun 30 '17 at 14:25
  • OK, just trying to help by getting as much relevant information as possible. Now you say it prints zero, one or two of the paths. *Is it always the same one or two?* – l0b0 Jun 30 '17 at 14:29
  • @l0b0 Different tests on different systems show different outputs and it's making the situation worse. On a Debian with kernel 4.9, the first run of the program prints two lines of output, on another Debian with kernel 4.9 it prints zero (nothing) lines after the first run, while in both systems it must return 5 lines. I suspect the first run behaves unpredictable. – user2808671 Jun 30 '17 at 14:34
  • 1
    Ah, so it's the *same* behaviour on a single system on every boot? That would be another clue. You might want to update the question with all the details you've dug out in this conversation, to make it easier for others to follow up on the original question. – l0b0 Jun 30 '17 at 14:42
  • @l0b0 Ok I edited the question. Also it's better that others check this scenario if they can and tell whether they get the same results. – user2808671 Jun 30 '17 at 14:52
  • Does `dmesg` show I/O errors? – Mark Wagner Jun 30 '17 at 18:13
  • 2
    @MarkWagner No. I filled a bug report to both gnu findutils and Debian backports mailing list. This seems very serious to me as the source of this problem may affect many other things though I don't know whether you guys agree with me. Anyways "find" is a very popular tool and its output must be reliable. – user2808671 Jun 30 '17 at 18:44
  • What is the time of creation for the [occasionally] missing files? Any chance that they were actually missing, and then got re-created somehow? Maybe something like a series of updates? – Jed Daniels Jun 30 '17 at 20:44
  • 1
    @JedDaniels The files existed from the beginning, the commands were run almost immediately after each other and as I said same thing happens after every reboot. – user2808671 Jul 01 '17 at 06:40
  • Maybe the first run happens before /lib is mounted. – Marcos Dione Jul 01 '17 at 10:05
  • 1
    Thanks, others mentioned similar thoughts too, but as I said all the files are accessible before running commands. I have no doubt that find is producing wrong (incomplete) result but I want to know why – user2808671 Jul 01 '17 at 10:45
  • 1
    @MarcosDione `/lib` needs to be on the root file system, otherwise you can't even run `mount`. – kasperd Jul 01 '17 at 11:34
  • Does it make any difference if you use the `-noleaf` flag for `find`? – kasperd Jul 01 '17 at 11:35
  • 1
    To clarify, when is "the first time"? Is it at any time after booting or at some specific point during the boot process? – Håkan Lindqvist Jul 01 '17 at 11:43
  • If you were to instead use eg `busybox find . -name "*systemd-resolved*"` at the time when `find` produces unexpected results, does that change anything? It might be a hint as to whether the problem is actually with `find` (seems like a longshot imo), or rather the environment. – Håkan Lindqvist Jul 01 '17 at 11:51
  • 2
    How is `/lib/systemd` mounted? What kind of filesystem is it? If it's a separate mount point, what *time* was it mounted? – Andrew Henle Jul 01 '17 at 12:21
  • 1
    @kasperd No difference – user2808671 Jul 01 '17 at 14:48
  • 1
    @HåkanLindqvist Any time after boot. I checked this after logging in to the system. Maybe this happens before those parts of kernel cache that corresponds to /lib/systemd path gets populated. But in general it happens after every reboot – user2808671 Jul 01 '17 at 14:51
  • 1
    @AndrewHenle it's not a separate device. It's inside the partition which is mounted at booting time on root path. Filesystem is Ext4 – user2808671 Jul 01 '17 at 14:58
  • 1
    @HåkanLindqvist On one of the systems which showed this error, I ran `busybox find . -name "*systemd-resolved*"` and it gave correct output with the first run. Now we're getting somewhere. What do you think? This may be due to implementation details between different versions of "findutils" but I still suspect the problem may result from somewhere beyond user space. – user2808671 Jul 01 '17 at 15:04
  • @user2808671 It sure sounds like caching is affecting the result. Have you tried comparing `strace` output from runs with different output? Use the `-o` flag to save the output to a file. – kasperd Jul 01 '17 at 18:06
  • 1
    @kasperd Yes it should be related to kernel caching. I uploaded output of strace for the first run at https://file.io/WS0Ked and the output of the second strace run at https://file.io/R3vZBa . You can take a look at these traces – user2808671 Jul 01 '17 at 19:45
  • 1
    @user2808671 Both of those links give me 404. – kasperd Jul 01 '17 at 21:32
  • @kasperd At time of writing the comment links where working! Anyways I uploaded on another server. Please check this https://ufile.io/js63a before it possibly gets expired! – user2808671 Jul 02 '17 at 04:34
  • @user2808671 That's an html file. What is the URL for the strace output? – kasperd Jul 02 '17 at 08:09
  • 1
    @kasperd That's the URL to _Download_ the file. I compressed both traces as gzip and uploaded there which you can download through the link I provided. The file name is trace.tar.gz – user2808671 Jul 02 '17 at 09:00
  • Since the missing files are all in `/lib/systemd`, do you see the same problem if you do `find /lib/systemd ...` instead of `find . ...`? – Barmar Jul 04 '17 at 18:39
  • 1
    as I see it, the difference in output depends on what directory your starting from. Suggest using: `find / -name "*systemd-resolved*"` so your starting from the 'root' directory. – user3629249 Jul 05 '17 at 22:51
  • 1
    @user3629249 What are you seeing exactly?! All those command are run in the root directory. I said everything is the same for executions except that one is run a few seconds after the first. – user2808671 Jul 06 '17 at 08:44
  • 1
    @Barmar Entering the path explicitly shows correct results (as expected) The problem is when "find" should traverse filesystem – user2808671 Jul 06 '17 at 08:52
  • 1
    @Barmar and all others in the discussion please read the edit. – user2808671 Jul 06 '17 at 08:54
  • I can not reproduce the failure in debian Stretch 4.9.0-3-amd64). – Tim Haegele Jul 06 '17 at 09:22
  • This reeks of a rootkit. Can you reproduce in a freshly-installed system ? Can you observe this behaviour if you boot the machine from a liveusb and run the liveusb find on its filesystem ? can you install and run sysdig, and compare its output with the one of strace ? – b0fh Jul 06 '17 at 10:51
  • I also can not reproduce the bug on a freshly-installed, fully-upgraded Debian 8 with kernel 4.9.0-3-amd64. I **strongly** suggest you run a boot-time `fsck` *and* to check for possible rootkits. – shodanshok Jul 06 '17 at 11:09
  • 1
    @shodanshok Can you please say what you exactly mean by "check for possible rootkits"? You say my systems might be infected by rootkits? – user2808671 Jul 06 '17 at 11:13
  • 1
    @b0fh The systems that I checked are: One of them my main operating system which I use and I have some security related projects on it. The others are Debian installations on virtual box from the same Debian iso image which I downloaded from the official site. The difference is that the ones installed on virtual box don't have the default kernel, instead I use built-from-source kernels for them. But the problem occurs for all of them even my main operating system with the kernel directly downloaded from Debian backports. Now do you think I may be a target of a rootkit attack? – user2808671 Jul 06 '17 at 11:20
  • Try with a fresh Debian installation on your Virtualbox host machine. I just did the same on a (on a KVM installation) and, as said, I found no strange behavior at all. – shodanshok Jul 06 '17 at 11:24
  • 1
    @b0fh The filesystem is Ext4 but inside a LUKS container. Also I've made various integrity checks scripts which are run at boot time. It's very strange and unlikely that someone could have installed (possibly) a kernel level rootkit on my systems. – user2808671 Jul 06 '17 at 11:24
  • @shodanshok How did you install it? Did you download an unstable image of stretch? The image I used is the stable jessie but after installation of the system, I had installed the backport kernel using apt-get – user2808671 Jul 06 '17 at 11:30
  • @user2808671 I downloaded [this image](https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/debian-live-9.0.1-amd64-mate.iso) and installed it in the classical way, with a single LVM partition and EXT4 filesystem – shodanshok Jul 06 '17 at 12:48
  • @shodanshok Would you check the version of findutils package of that image by running `dpkg --status findutils` inside the installed system? – user2808671 Jul 06 '17 at 12:56
  • @user2808671 I have 4.6.0+git+20161106-2, which seems to be the only available version for [Debian 9 Stable](https://packages.debian.org/search?keywords=findutils&searchon=names&suite=stable&section=all) Are you using an old Unstable/Testing version? – shodanshok Jul 06 '17 at 13:08
  • 1
    @shodanshok Debian 9.0 was released June 17th, 2017, and I have installed my OS before that time so yes, I had downloaded Debian 8 stable and upgraded to latest kernel using apt-get and backport repositories and always kept my system up to date. – user2808671 Jul 06 '17 at 13:16
  • @user2808671 I installed a Debian 8.8 with backports, kernel 4.9.0-0.bpo.3-amd64 and findutils 4.4.2-9+b1 and I can not replicate your problem – shodanshok Jul 06 '17 at 15:42
  • @shodanshok This is why I call it very strange. Thanks for taking time to try to reproduce the problem and reply in the comments. – user2808671 Jul 06 '17 at 17:26
  • What happens if you use `find / ...` instead of `find . ...`? Also, your first trace (in the gzipped download) looks incomplete, because it never outputs the file names it found. – Moshe Katz Jul 06 '17 at 19:57
  • 1
    @MosheKatz No difference using `find /` or `find .` . The file is not incomplete, the reason there's no output is that first run actually did not return anything. I did mention this in question text and other comments too. – user2808671 Jul 06 '17 at 20:58
  • You didn't say that it didn't find "*anything*", you said it found two of the five files. – Moshe Katz Jul 06 '17 at 21:11
  • 1
    @MosheKatz In the question text I said _The command returns 0 or two lines as output for the first run_ and in response to another user I said _on another Debian with kernel 4.9 it prints zero (nothing) lines after the first run_ – user2808671 Jul 06 '17 at 21:15
  • Ok, I see. Can you also get an strace of when it provided two? – Moshe Katz Jul 06 '17 at 21:37
  • @MosheKatz I'm sorry no, I removed the offending binary. – user2808671 Jul 06 '17 at 21:42
  • per your description, the Debian 8 (jessie) has a bug. – user3629249 Jul 06 '17 at 23:18

1 Answers1

4

The default version of findutils which is installed on Debian 8 is 4.4.2 and this is the newest version on jessie repositories. I download latest version (4.6.0) of findutils source code and built the binaries from source. Then I did same tests and "find" command showed correct output for the first run.

Then I downloaded findutils version 4.4.2 source code from gnu archive and compiled it. The same problem happened for the compiled find command. So this problem is not happening with findutils 4.6.0.

But I still don't know why some users don't get same results using findutils 4.4.2 (the default version of the utility installed on Debian), and don't know why Debian should still be released with this old version of findutils and possibly other Linux utilities and cause this problematic situation. And the last thing is that the exact technical reason of what strangely happened is still unknown which is not desirable. Because I'm not sure whether there's something worrisome in my OS environment.

user2808671
  • 183
  • 10