2

I stupidly created a reasonably deep tree (4 folders deep) with the leaf folders containing a large amount of smallish files (over 1 million files around 200kb to 1mb in size). In total the count would be over 80 GB of files. This was created in the /tmp/ directory of a Ubuntu Server 14.04 box running inside Hetzner (relevant to come up with way to connect to it possibly).

From memory running a "find . | wc -l" took over 12 hours to run.

I then needed to reboot the server. This is not normally an issue as it has been rebooted multiple times over the last few weeks without issue. No config changes have taken place nor additional software installed.

In this case however it is not coming back online. I can ping the box but I cannot ssh or connect to any port I would expect to come back online.

I suspect the issue is that the script that mounts and clears tmp on boot /etc/init/mounted-tmp.conf is struggling to clear the files, causing the box to be stuck in the init boot phase.

I don't suppose there is any way to speed this process up? I know if I could disable the tmp clean somehow I could use the rsync trick to clean the folders but I have no idea how to even do this.

Ben Boyter
  • 171
  • 4
  • If this is the problem you can' speed it up. Do you have console access ? Are there any messages there ? – user9517 Oct 02 '15 at 05:54
  • The speed is a problem. I would like to have it back up ASAP. No messages. However I am not even sure if that is the problem at this point. I suspect it may be but not 100% sure. – Ben Boyter Oct 02 '15 at 05:58

2 Answers2

2

Found a solution.

Booted the machine using a rescue system as offered by hetzer http://wiki.hetzner.de/index.php/Hetzner_Rescue-System/en

Then mounted the filesystem and used the tips from here https://unix.stackexchange.com/questions/37329/efficiently-delete-large-directory-containing-thousands-of-files on how to delete the files in a more efficient manner.

Rebooted and now all is fine.

Community
  • 1
Ben Boyter
  • 171
  • 4
1

I see you answered your own question, but here's my 2 cents.

What about this booting into single user mode? Do you have access to grub? On boot, edit the boot parameters to include the "single" option, then boot straight to a root command prompt. From there you can clean up your files.

Ryan Babchishin
  • 6,260
  • 2
  • 17
  • 37
  • Did not have the ability to do that through the tools they provided. That would work well on a home system though I would think. – Ben Boyter Oct 12 '15 at 23:28