50

I have an pages.txt file with 100 URLs inside. I want to check them one by one and fail on the first problem. This is what I'm doing:

cat pages.txt | xargs -n 1 curl --silent \
  --output /dev/null --write-out '%{url_effective}: %{http_code}\n'; echo $?

Exit code is 1, but I see it only when the entire file is done. How to stop earlier, on the first problem?

yegor256
  • 102,010
  • 123
  • 446
  • 597
  • Do you really need it to be done in one command, or is a script ok? – Sir Celsius Oct 21 '14 at 10:36
  • 1
    I want to use `xargs` because of its `-P` option – yegor256 Oct 21 '14 at 14:10
  • If you're looking for parallel execution, you may also want to investigate GNU `parallel`... It has a lot more flexibility than `xargs` simplistic model... – twalberg Oct 24 '14 at 19:36
  • 1
    Just a note: `xargs` comes installed on most systems. `parallel` always takes an extra installation step. In case that's important to someone. – 425nesp Mar 05 '20 at 02:34

2 Answers2

75

General method

xargs -n 1 sh -c '<your_command> $0 || exit 255' < input

Specific case

xargs -n 1 sh -c 'curl --silent --output /dev/null \
    --write-out "%{url_effective}: %{http_code}\n" $0 || exit 255' < pages.txt

Explanation

For every URL in pages.txt, executes sh -c 'curl ... $0 || exit 255' one by one (-n 1) forcing to exit with 255 if the command fails.

From man xargs:

If any invocation of the command exits with a status of 255, xargs will stop immediately without reading any further input. An error message is issued on stderr when this happens.

whoan
  • 8,143
  • 4
  • 39
  • 48
  • 1
    Perfect. I have a script to checksum validate files. Since I'm typically doing a whole disk at once that is way too many arguments for a simple shell script loop. Typically though, if I press control-c because I see an error or such, xargs keeps on spawning the rest of the processes. Putting in the exit 255 resolves that issue. It also means if an IO error that takes the drive offline, the everything will stop at the file that caused the error. – Bill Apr 18 '20 at 13:10
3

I haven't found a way to do what you ask for with xargs, but a loop with read might be what you are looking for.

while read URL; do
  curl --silent \
    --output /dev/null --write-out '%{url_effective}: %{http_code}\n' $URL;
  RET=$?;
  echo $RET;
  if [ $RET -ne 0 ]; then break; fi
done < pages.txt
Benjamin Loison
  • 3,782
  • 4
  • 16
  • 33
davrieb
  • 559
  • 2
  • 7
  • 2
    The problem here is that I can't run them in parallel, while `xargs` has a `-P` option for that – yegor256 Oct 21 '14 at 14:02
  • @yegor256 Are you saying that you want to launch processes in parallel, but stop when the "first one" exits with an error? Which process is the "first one" that executes/fails? What happens to those in-process when one of them fails? – Christopher Schultz Jul 13 '17 at 14:38
  • Normally if I use xargs, it is because I have way too many arguments for the command line. The tip for exit 255 works for me. I also takes care of the problem of aborting correctly when I press control-c. – Bill Apr 18 '20 at 13:07