1

I have 3 AWS EC2 machines, using each 500G of EBS storage. On them I have SolrCloud search engine running on Jetty, and serving lots of queries. In time the jetty logs have grwn hugely, up to ~200G in size. I would like to remove that, but without giving up availability of my solr system. If i do rm -rf logfile then CPU wait goes up to 99% and the nodes on that machines are not responding anymore, until the load decreases.

Is there a way to remove those log files without this huge load increase? Thanks

Later Edit: So, i've found online 4 ways of emptying big files, but there is no comparation between, so i'm not sure which satisfies my problem. Pros and cons for each of them?

1.> largefile.txt
2. truncate -s 0 {filename.txt}
3. logrotate
4. cat /dev/null > largefile.txt
maephisto
  • 131
  • 4
  • `: > logfile` will zero the file... – ewwhite Jan 07 '14 at 13:49
  • @ewwhite thanks! would this be light on the CPU? – maephisto Jan 07 '14 at 13:52
  • @ewwhite i've also found this articlehttp://www.cyberciti.biz/faq/truncate-large-text-file-in-unix-linux/ but it says nothing on CPU performance – maephisto Jan 07 '14 at 13:56
  • Test these approaches in *your* environment. – ewwhite Jan 07 '14 at 14:16
  • On a Unix/Linux system truncating a file (by any of the means described) still leaves the data on the disk - and the application with the open handle will continue to write to the shadow file (that's why logrotate uses a script which usually includes sending a signal to the server to close and re-open it's log files). – symcbean Jan 07 '14 at 16:17

2 Answers2

1

You could try limiting the amount of CPU that the rm process can use. It will take longer to run but it won't swamp your system. Here is an article talking about that using a program called cpulimit.

Eric
  • 1,383
  • 3
  • 17
  • 34
1

the problem might not even be the CPU, but IOLOAD.

i had the issue with a few > 20GB sized Logfiles some time ago.

basically what i did to reduce the impact of the removal:

nice -n19 ionice -c3 rm LOGFILE

ionice however might not work correctly for your hardware setup and/or io scheduler settings.

According to Stu Thompson you might need the CFG scheduler for a working ionice command.

ionice not working for you may have been because you were running with the incorrect scheduler. It apparently only works with the CFQ scheduler. – Stu Thompson Oct 31 '11 at 12:28

See this for the ionice CFG scheduler issue

Community
  • 1
Dennis Nolte
  • 2,881
  • 4
  • 27
  • 37