Is there any way reduce the I/O wait times in linux jobs?

Question

I am running a multiple parallel file handling process (minimum 200 processes) where each process is reading a logs of varying size(0-50mb) to capture a real time data of the logs .I am running my jobs on 16 cores and 8 gb ram Linux machine.

But there is a huge lag between the cpu and elapsed time process of the files.

Can you please explain why there is such lag b/w cpu and elapsed time with some examples?Also is there any method to reduce the wait times of these processes.

Run fewer jobs at a time to reduce the contention on your I/O system, or add more memory for more effective caching. — twalberg, Mar 31 '17 at 19:06
@twalberg it's not possible for me to run fewer jobs because I have to capture real time data — vikas chib, Apr 01 '17 at 08:19
Can you use local disk during job execution, and then copy the output to shared storage at the end of the job? Local disk has much better performance than shared disk. — Michael Closson, Apr 01 '17 at 17:22

score 0 · Answer 1 · answered Mar 31 '17 at 15:27

What distro are you running? I ask because one way to approach this would be to use a different I/O scheduler but your distro may not support changing your scheduler. Certain I/O schedulers are best for different filesystems and architectures so I would recommend looking here to get an idea of different schedulers. As for the discrepancy you're experiencing, I honestly have no idea what that's happening.

Also, I'm assuming the log files are located on an SSD?

log files are created on HDD and I can not use I/O scheduler — vikas chib, Mar 31 '17 at 15:39

Is there any way reduce the I/O wait times in linux jobs?

1 Answers1