Why does xxd add chars?

Question

I am trying to reverse an od command from a system where I have no hexdump or base64 tools.

I do this like that (of course, in reality, the encoding takes place at the "small" system, the decoding is done on my workstation, but to test it, I try the whole way in one line first):

echo TEST | od -tx1 | xxd -r

Of course, echo TEST is just a placeholder here for eg. cat test.bmp or anything else.

> echo TEST
TEST

> echo TEST | od -tx1
0000000 54 45 53 54 0a
0000005 

> echo TEST | od -tx1 | xxd -r
TEST

That looks right, but it is different, as we can see here if we give it to od again:

> echo TEST | od -tx1 | xxd -r | od -tx1
0000000 54 45 53 54 0a 00 00 00
0000010

Why does xxd -r add those 00s?

It seems to have something to do with the second line of `od`'s output. Try `echo -n TEST | od -tx1 | head -1 | xxd -r | od -tx1` — Shawn, Jul 19 '19 at 08:49
No, xdd adds it: `echo TEST | od -tx1 | xxd -r | wc -c` equals to 8. — Bayou, Jul 19 '19 at 08:50
And compare `echo -n TEST | xxd | xxd -r | od -tx1`. Looking at the man page, `xxd -r` expects `xxd` style hex dumps, not an `od` one. The difference might also have something to do with it. — Shawn, Jul 19 '19 at 08:52
`echo TEST | od -tx1 | head -1 | xxd -r | wc -c` is 5. It's something about that second line that od prints not matching the input format that `xxd -r` expects. The old saying is 'Garbage In, Garbage Out'. — Shawn, Jul 19 '19 at 08:55
also the address are in octal with od (except `-Ax`) whereas hexa in xxd. compare `printf 'hello\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0world' | od -Ax -tx1` — Nahuel Fouilleul, Jul 19 '19 at 09:31

Shawn · Answer 1 · 2019-07-19T09:35:06.257

You're getting those three nul bytes because of xxd -r trying and failing to parse input that's in a different format than what it expects. od -tx1 adds an extra line with an offset but no data bytes. Plus the offsets in xxd have a colon after them, and are printed with a different width, and printable bytes are displayed as well as the hex dump, and possibly in a different base... Something about that doesn't play well with xxd, and it adds the extra bytes as a result.

Examples:

$ echo TEST | xxd
00000000: 5445 5354 0a                             TEST.
$ echo TEST | xxd | xxd -r
TEST
$ echo TEST | xxd | xxd -r | xxd    
00000000: 5445 5354 0a                             TEST.
$ echo TEST | xxd | xxd -r | od -tx1
0000000 54 45 53 54 0a
0000005
$ echo TEST | od -tx1 | head -1 | xxd -r | od -tx1 
0000000 54 45 53 54 0a
0000005

See how they're not present when giving xxd -r its expected xxd style input? And how they're not there when you prune that extra line from od's output? Don't mix and match incompatible data formats.

score 0 · Accepted Answer · answered Jul 19 '19 at 10:35

It seems to work if I remove the offsets at all:

> echo TEST | od -tx1 -An
 54 45 53 54 0a
> echo TEST | od -tx1 -An | xxd -r -p
TEST
> echo TEST | od -tx1 -An | xxd -r -p | od -tx1 -An
 54 45 53 54 0a

Bingo! Note the extra " " in front of the bytes. It seems to have no effect.

Why does xxd add chars?

2 Answers2