5

I am currently working with a large data set where even the file format conversion takes at least an hour per subject and as a result I am often unsure whether my command has been executed or the program has frozen. I was wondering whether anyone has a tip to how to follow the progress of the commands/scripts I am trying to run in linux?

Your help will be much appreciated.

Ja͢ck
  • 170,779
  • 38
  • 263
  • 309
user1442363
  • 800
  • 1
  • 10
  • 18
  • 3
    Which commands? Some of them may have options for verbose output. – Thom Smith Jun 11 '12 at 13:06
  • Shell scripts can be run by `sh -x` instead of `sh` to show the commands prior to executing them. – Fred Foo Jun 11 '12 at 13:08
  • If you produces a log or any kind of file, you can run `tail -f` on this file. – Denys Séguret Jun 11 '12 at 13:09
  • 1
    If all else fails, you can spawn `top` in another console and monitor the CPU usage of your command. – Frédéric Hamidi Jun 11 '12 at 13:18
  • The command I am currently waiting output from is `find . -mindepth 4 -type d -exec tractor -d -r -b preproc RunStages:1 Interactive:FALSE '{}' \;` – user1442363 Jun 11 '12 at 13:43
  • Possible duplicate of [How can I make a progress bar while copying a directory with cp?](https://stackoverflow.com/questions/7128575/how-can-i-make-a-progress-bar-while-copying-a-directory-with-cp) – jww Oct 08 '18 at 00:53

5 Answers5

4

In addition to @basile-starynkevitch answer,
I have a bash script that can measure how much file did you processed in percents.

It watch into procfs get current position from fd information (/proc/pid/fdinfo), and count this in percents, relative to total file size.

See https://gist.github.com/azat/2830255

curl -s https://gist.github.com/azat/2830255/raw >| progress_fds.sh \
&& chmod +x progress_fds.sh

Usage:
./progress_fds.sh /path/to/file [ PID]

Сan be useful to someone

azat
  • 3,545
  • 1
  • 28
  • 30
  • I took your script as an inspiration and extended the mechanism a little. My Python script can be viewed at https://code.activestate.com/recipes/578882-monitor-progress-of-file-descriptors-of-another-pr/ . – Alfe May 19 '14 at 11:28
  • Yeah, @Alfe nice improvement, I will look into this! Thanks! – azat Jun 06 '14 at 22:06
  • concise and extremely useful, this should have many more up votes – pyRabbit Mar 31 '16 at 15:34
3

If the long-lasting command produces some output in a file foo.out, you could do watch ls -l foo.out or tail -f foo.out

You could also list /proc/$(pidof prog)/fd to find out the opened files of some prog

Basile Starynkevitch
  • 223,805
  • 18
  • 296
  • 547
2

You can follow the syscalls of a program by using strace, which will enable you to follow the open calls.

thiton
  • 35,651
  • 4
  • 70
  • 100
1

You can use verbose output, but it will slow things down even more.

Denis
  • 105
  • 1
  • 10
1

I guess there can't be a general answer to that, it just depends on the type of program (that doesn't even has to do anything with Linux, see the "halting problem").

If you happen to use a pipe during the conversion I find the pv(1) tool pretty helpful. Even if pv can't know the total size of the data it helps to see if there is actual progress and how good the datarate is. It isn't part of most standard installations though and probably has to be installed explicitly.

David Ongaro
  • 3,568
  • 1
  • 24
  • 36