1

I need to replace several URLs in a text file with some content dependent on the URL itself. Let's say for simplicity it's the first line of the document at the URL.

What I'm trying is this:

sed "s/^URL=\(.*\)/TITLE=$(curl -s \1 | head -n 1)/" file.txt

This doesn't work, since \1 is not set. However, the shell is getting called. Can I somehow push the sed match variables to that subprocess?

Nikolai Prokoschenko
  • 8,465
  • 11
  • 58
  • 97
  • Possible duplicate of [sed - pass match to external command](http://stackoverflow.com/questions/11719011/sed-pass-match-to-external-command). Although this question is older than the proposed duplicate, that duplicate does have working answers, while for this one even the accepted answer doesn't work. – Ruslan Jun 25 '15 at 09:56

4 Answers4

4

The accept answer is just plain wrong. Proof:

Make an executable script foo.sh:

#! /bin/bash

echo $* 1>&2

Now run it:

$ echo foo | sed -e "s/\\(foo\\)/$(./foo.sh \\1)/"
\1
$

The $(...) is expanded before sed is run.

e40
  • 397
  • 1
  • 11
  • you ahve too many backslashes, and the capturnign group is empty.. echo foo | sed "s/\\(foo\\)/$(./foo.sh \\1)/" – Eddie Nov 12 '12 at 20:18
  • The parameter to the script is still `\1` instead of `foo`. The number of backslashes does not change anything in this case: both `echo afoos | sed -e "s/\(foo\)/x/"` and `echo afoos | sed -e "s/\\(foo\\)/x/"` output `axs`. This answer is right. – Arsen7 Jun 19 '15 at 14:12
  • `(echo "a a a a a a") | sed "s/a/$(echo $RANDOM)/g"` -- this is a probably simpler to understand example that replaces each instance of `a` with a random number (using `$RANDOM`), but all replacements are the same number because the `$(...)` shell expansion happens before the `sed` execution and only one random number was generated in the first place. – Alexander Klimetschek Nov 08 '17 at 20:53
3

So you are trying to call an external command from inside the replacement pattern of a sed substitution. I dont' think it can be done, the $... inside a pattern just allows you to use an already existent (constant) shell variable.

I'd go with Perl, see the /e option in the search-replace operator (s/.../.../e).

UPDATE: I was wrong, sed plays nicely with the shell, and it allows you do to that. But, then, the backlash in \1 should be escaped. Try instead:

sed "s/^URL=\(.*\)/TITLE=$(curl -s \\1 | head -n 1)/" file.txt
leonbloy
  • 73,180
  • 20
  • 142
  • 190
  • It works, try it with $(date) inside the replacement pattern. – Nikolai Prokoschenko May 01 '10 at 17:09
  • Ha, sed is more powerfull than I believed. Then, try escaping the backlash with \\1 – leonbloy May 01 '10 at 17:35
  • 2
    I would have expected that to evaluate the `$(...)` expression *first*, before invoking sed; so curl would try to download the URL "`\1`". And some of your other backslashes surely got eaten, since you were using double quotes. – zwol Aug 03 '10 at 01:57
  • oh no, this would be exactly what I was searching for over an hour, only the old sed v4.1.5 from 2006 does not seem to support this:( (tried both $() syntax and backticks ``) – Gregor Jan 18 '12 at 18:29
  • Thanks, i did search for an hour, this is exactly what i want. – wukong May 22 '13 at 15:34
1

Try this:

sed "s/^URL=\(.*\)/\1/" file.txt | while read url; do sed "s@URL=\($url\)@TITLE=$(curl -s $url | head -n 1)@" file.txt; done

If there are duplicate URLs in the original file, then there will be n^2 of them in the output. The @ as a delimiter depends on the URLs not including that character.

Dennis Williamson
  • 346,391
  • 90
  • 374
  • 439
  • This, unlike the accepted answer, seems like it *ought* to work. Well, except for the backslashes inside the double quotes. – zwol Aug 03 '10 at 01:58
0

Late reply, but making sure people don't get thrown off by the answers here -- this can be done in gnu sed using the e command. The following, for example, decrements a number at the beginning of a line:

echo "444 foo" | sed "s/\([0-9]*\)\(.*\)/expr \1 - 1 | tr -d '\n'; echo \"\2\";/e"

will produce:

443 foo
blackghost
  • 1,730
  • 11
  • 24