0

[edited to add -b, which I had already tried to no effect]

I had a file with a column of numbers in parentheses I wanted to sort on, something like this:

x (10)
x (11)
x (1)
x (2)

I thought that sort -b -k 2.2n would work. But it didn't. But I discovered that sort -b -k 2.2 -n does work, generating the desired output

x (1)
x (2)
x (10)
x (11)

Can anyone explain why? I know that -n treats all columns as numeric (not just the selected ones), but I'm surprised that makes a difference here. I really thought that -k 2.2n would sort the second column numerically, starting at the second position, and I don't understand why it didn't work. (Though sort's nuances have certainly eluded me before.)

This is "sort (GNU coreutils) 5.93", if it makes a difference. [And I later tried it on another machine with coreutils 8.5 and saw the same results.]

Community
  • 1
  • 1
Steve Summit
  • 45,437
  • 7
  • 70
  • 103

1 Answers1

3

Coreutils 5.93 is really old. The newer versions have a nice --debug option that shows you how your field selectors apply to each line.

The problem you're having is that sort field-splitting by default includes the whitespace between fields as part of the following field. So in your case, the second field is " (1)" with a leading space so -k2.2 is selecting a substring starting with a parenthesis, which then fails to be recognized as a number.

You can fix this by adding the b flag. sort -k2.2nb or sort -k2.2 -n -b should work.

On a more recent coreutils (8.23), I can't reproduce your behavior of sort -k2.2 -n - it behaves the same as sort -k2.2n, failing to match the numbers and falling back on the "whole line string comparison" sort.

UPDATE (3): Now I have reproduced your result, in versions 5.93, 8.13, and 8.23. And there is a definite explanation for it.

Both of my suggestions (sort -k2.2nb and sort -k2.2 -n -b) work. Your sort -b -k2.2n doesn't. In a version that supports --debug, it says:

$ printf '%s\n' 'x ('{1,2,10,11}')'|sort -k 2.2n -b --debug
sort: using simple byte comparison
sort: leading blanks are significant in key 1; consider also specifying 'b'
sort: key 1 is numeric and spans multiple fields
sort: option '-b' is ignored
x (1)
  ^ no match for key
_____
x (10)
  ^ no match for key
______
x (11)
  ^ no match for key
______
x (2)
  ^ no match for key
_____
$

The option '-b' is ignored is the explanation of what's happening. As soon as you attach one flag (the n) to the a key specification given with -k, all globally-specified flags are ignored for that key.

The GNU info documentation for coreutils makes this point clearly (I marked the important part in bold):

The following options affect the ordering of output lines. They may be specified globally or as part of a specific key field. If no key fields are specified, global options apply to comparison of entire lines; otherwise the global options are inherited by key fields that do not specify any special options of their own. In pre-POSIX versions of ‘sort’, global options affect only later key fields, so portable shell scripts should specify global options first.

(and -b is in the list that follows.)

The man page wording is less clear:

KEYDEF is F[.C][OPTS][,F[.C][OPTS]] for start and stop position, where F is a field number and C a character position in the field; both are origin 1, and the stop position defaults to the line's end. If neither -t nor -b is in effect, characters in a field are counted from the beginning of the preceding whitespace. OPTS is one or more single-letter ordering options [bdfgiMhnRrV], which override global ordering options for that key. If no key is given, use the entire line as the key.

You could easily interpret that as meaning that each single-letter option overrides the global option of the same letter only. But you'd be wrong...

The POSIX definition is pretty clear (I bolded the important piece again):

The following options shall override the default ordering rules. When ordering options appear independent of any key field specifications, the requested field ordering rules shall be applied globally to all sort keys. When attached to a specific key (see -k), the specified ordering options shall override all global ordering options for that key.

The only slightly unclear part about it is that -b doesn't appear in the list immediately following that paragraph. There's a single sentence after that list, introducing another list containing the -b and -t options, so you might argue that -b isn't within the scope of the "override all global ordering options" rule.

After all, -b isn't an ordering option, but a field splitting option, and when you attach it to a key definition you can apply it separately to the beginning, the end, or both. So it is not like any of the other options. But at least in the GNU implementation, it does follow the "override all global ordering options" rule.

  • Thanks. I forgot to mention that in the real script that this question presented a simplified version of, I *was* using `-b`, and it didn't help. – Steve Summit Jul 17 '15 at 13:47
  • Can you really not reproduce it? I just tried it on another machine, with coreutils 8.5, and saw the same results (with `-b`). – Steve Summit Jul 17 '15 at 13:53
  • Does 8.5 have `--debug`? –  Jul 17 '15 at 13:56
  • No, it does not -- I was just trying that. Very strange; it sounds like a dandy option, because `sort`'s definitions of fields are typically so different from most other 'nix utilities. – Steve Summit Jul 17 '15 at 13:57
  • Thanks for all that legwork, but I'm afraid you missed my comment and edit where I said that I _had_ been using `-b` all along. I think that's why you can't reproduce my result, because you're not using it. – Steve Summit Jul 18 '15 at 10:10
  • Nice work. There is another question where this sort of issue came up, but there is more and better information in this answer. The `--debug` option is nice too; I ran into all sorts of problems working out that this sort of mess was the cause of my trouble —and the blank was counted, completely counter-intuitively, as part of the field. That is a weird aspect of the POSIX specification. There must be precedent (presumably strong precedent) for it, but it seems really odd to me. I think it's safe to say that, were it up to me, the separator would not be counted at the start of a field. – Jonathan Leffler Jul 18 '15 at 13:44
  • Wow. Thanks. I had given up on this, and I was about to file a bug report, because I hadn't seen your "UPDATE (3)". (For some reason the SO machinery didn't notify me.) I keep thinking I understand `sort`'s idiosyncrasies, and it keeps tripping me up with ever-more subtle ones. Thanks very much for researching this; I never would have figured this out. – Steve Summit Jul 19 '15 at 18:04