0

I want to hash the contents of 4 million JPEG files taking up 32TB of a 48TB RAID6 partition in CentOS 7. It's a Hetzner SX131 10 x 6TB box.

Having run a test on 10,000 files I think the full job is going to take 72 hours.

Would you just set it and forget it, or is it prudent to add (say) a 5 second pause every minute?

Codemonkey
  • 1,086
  • 4
  • 19
  • 41
  • 1
    What are you hoping to gain by adding the pause? Surely that will only make the job take longer. If you're concerned about soaking up the entire CPU, run the job under `nice`. `man nice` for more details. You may also be able to restrict the job to run on only a certain portion of the available CPUs. But without knowing what the goal is that you're after, it's hard to suggest appropriate tactics. – Jim L. Nov 05 '19 at 00:53
  • Not worried about the CPU, the box has nothing else to do at the moment. I'm worried about the hard drives; heat buildup etc. I've just checked iotop and it's sitting at 100MB/s when my script is running. Am I worrying about nothing? I'm far from an expert in this field, as you can tell, and just want to check I'm not being stupid. – Codemonkey Nov 05 '19 at 00:55
  • 1
    I would run it under `nice` regardless. If the box truly has nothing else to do, `nice` won't slow things down. And I wouldn't worry about a high duty-cycle on the hard drives. If you feel up to it, you could run a script in parallel that queries the hard disk temperatures every 60 seconds via SMART or whatever interface is available, and `printf` those readings to a file. That will give you a temperature history of the job to indicate whether your temperature concerns were justified. Record the CPU temp, also, if you can. It may warm up more than the disks do, but still no worries. – Jim L. Nov 05 '19 at 01:03
  • Thanks Jim; from what I recall the RAID controller makes getting at the SMART data troublesome, but I'll have a look. – Codemonkey Nov 05 '19 at 01:05

1 Answers1

2

You should be fine, providing as you say its the only thing the system is doing.

From the POV of the CPU you might want to run multiple processes simultaneously to speed things up if that's a bottleneck. More relevant to your question though - a CPU will throttle and self protect if it gets to warm, so the system will protect itself just fine from a CPU POV.

From a RAM POV I doubt this will make a difference.

From an HDD POV, if drives are going to die they are likely to do so anyway. I doubt the slight reduction of heat will make a significant difference. Also bear in mind that startup is heavier on wear and tear them just running a process. If it were me I would not worry, or put a couple of fans blowing over the disks.

Doing high load task for a few days is well within the ability of any decent server - I'd just let it run.

davidgo
  • 6,222
  • 3
  • 23
  • 41
  • Thanks @davidgo - the server is in a datacentre in a different country to me so rocking up to add some fans probably isn't an option... But I guess I should be able to assume that their cooling will be up to the job anyway. Then again, Hetzner is a budget brand...! – Codemonkey Nov 05 '19 at 01:06
  • Paralleling the script to use the other cores occurred to me, but given that just one core is putting 100MB/s read on the array adding other cores into the mix feels unnecessary. This is a one time operation to generate hashes before I transfer these files to another location, so I'm in no great rush. – Codemonkey Nov 05 '19 at 01:09