2

I have a CentOS 6.9 server.

The full /etc directory was under version control.

I accidentally ran 'git rm -R' on it. The server stopped working right away, my ssh session got lost.

I managed to have an ip console attached to the server, and booted from usb with a live linux, I ran 'git reset --hard'.

Now index/working dir is clean. I rebooted the system, but it still doesn't work. it responds to ping but i can't ssh to it still.

On the IP console, I can't even log in, after keying any user and it asks for user again.

From the recovery linux, I checked the errors in /var/log/boot.log -- first error is regarding iptables-restore

error while loading shared libraries libip4tc.so.0 : cannot open shared object file: No such file or directory

Next is with rsyslog.d

CONFIG ERROR: could not interpret master config file '/etc/rsyslog.conf'

The file is clearly there if I check from the recovery live linux. I suspect some permissioning issue but no idea really.

Any help appreciated. Thanks.

stvn
  • 23
  • 3

2 Answers2

3

The ownership and permissions are probably wrong. Since you weren't tracking these, you'll have to fix them manually, for every single file.

By itself, git doesn't track ownership and permissions of files. That's not what it's for. This metadata was not tracked at all and was therefore lost.

This is what systems like etckeeper are for. They wrap around git to keep /etc in version control, while also keeping the metadata that git doesn't, and restoring it if necessary. To prevent this happening in future, consider using etckeeper or a similar tool.

You may also want to institute some sort of backups. You currently don't appear to have any sort of backups, and this is a recipe for disaster. You've been lucky to escape with as little data loss as you have. You may not be so lucky next time.

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
  • Thanks! Actually I just relaized I do have a regular backup still in place (which I wanted to retire with git some time back) But strangely enough I get the same error. the backup was taken `tar jcvf /etc` , I restored with `tar xf` . I'm missing something obvious I hope – stvn Mar 15 '18 at 18:11
  • Turned out my backups was missing the se context - my box is up now. Thanks for the help. – stvn Mar 15 '18 at 19:50
  • You can restore default SELinux contexts with `restorecon`, e.g. `restorecon -rv /`. That should fix up most things. – Michael Hampton Mar 15 '18 at 21:24
  • List of `git` limitations: https://git.wiki.kernel.org/index.php/ContentLimitations to see what is missing when used as a backup tool. – Patrick Mevzek Mar 16 '18 at 23:47
  • Also see `rpm --setperms` to fix some permissions. This wont catch things that were generated after the fact however. Backups are better as Michael mentioned. – Aaron Mar 19 '18 at 17:50
0

Yes, you have a permission and file ownership problem. Unfortunately git only stores the content and executable bit of a file, so the read/write permissions, user and group ownerships, and ACLs if any are lost when you restore from git reset --hard. That's more than enough to break your host when the directory being restored is /etc.

Your best hope of recovery is from a full filesystem backup, which will include all of the file ownerships and permissions.

If you don't have one of those, you can try rebuilding the ownership and permissions manually by comparing them to a similar CentOS host. But it might end up being less work to reinstall the OS from scratch. Sorry.

Andrew Schulman
  • 8,811
  • 21
  • 32
  • 47
  • Thanks! That is helpful - actually I just relaized I do have a regular backup still in place (which I wanted to retire with git some time back) But strangely enough I get the same error. the backup was taken `tar jcvf /etc` , I restored with `tar xf` . I'm missing something obvious I hope – stvn Mar 15 '18 at 18:14
  • Turned out my backups was missing the se context - my box is up now. Thanks for the help. – stvn Mar 15 '18 at 19:55