bash - hash binary contents of variable without creating a file

Question

I am trying to obtain the hash of the contents stored in a variable created from a curl statement without outputting the curl into a file.

Basically I am trying to avoid:

curl website.com/file.exe >> tempfile.bin
md5sum tempfile.bin

The above provides the correct md5 hash. My approach (below) seemed to have work when testing plain text files, however when I attempted to download a binary and save it to a variable the hash was different than when I saved it to a file.

My attempt:

binary=$(curl website.com/file.exe)
echo $binary | md5sum

I think I may be missing a flag, or perhaps echo may not be the best way to do it. The important part of this challenge is not writing a file to disk, yet achieving the same md5 hash as if it were written to disk.

@Onitsoga You most certainly don't want `-e`. Did you quote the variable? — Biffen, Jan 30 '16 at 21:06

Benjamin W. · Accepted Answer · 2018-08-30T13:42:00.460

3

To also skip the step of using a temp variable, you can use process substitution:

md5sum <(curl website.com/file.exe)

or pipe to md5sum directly:

curl website.com/file.exe | md5sum

edited Aug 30 '18 at 13:42

answered Jan 30 '16 at 21:04

Benjamin W.

46,058
19
106
116

Don't really need a process substitution. `curl | md5sum` would do as well (and more portable too). – gniourf_gniourf Jan 30 '16 at 21:07
@gniourf_gniourf Added! – Benjamin W. Jan 30 '16 at 21:09
This seemed to work perfectly! Haha and its so simple! – Onitsoga Jan 30 '16 at 21:17
Actually I forgot to mention I would need to preserve the binary in the variable for later use. So I still need to rely on an echo? – Onitsoga Jan 30 '16 at 21:30
`var=$(curl ...; echo x); var=${var%x}`. The command substitution will strip any trailing newlines from the output of the command, so you need a sentinel value to ensure you don't lose any final bytes. – chepner Jan 30 '16 at 22:35

bacongravy · Answer 2 · 2016-01-31T07:36:17.677

1

The bash shell doesn't handle raw binary data well, as you've experienced. To accomplish your goal, you will need to encode the file contents into a text format when you read them into the bash variable, and decode them when you write them out.

For instance, if you have the base64 tool, you can use it to re-implement your example like this:

encoded=$(curl website.com/file.exe | base64)
echo "$encoded" | base64 --decode | md5sum

If you later want to save the data to a file named $output, you could do it like this:

echo "$encoded" | base64 --decode -o "$output"

edited Jan 31 '16 at 07:36

answered Jan 30 '16 at 21:14

bacongravy

893
8
13

If this works this is more along the lines of what I need. However when I run this I receive the error "base64: invalid input" – Onitsoga Jan 30 '16 at 21:42
After a few more tries this worked perfectly! Thanks! I had to echo the variable in quotes. So for my example: `binary=$(curl website.com/file.exe | base64); echo "$binary" | base64 --decode | md5sum` This gave me the propper md5sum – Onitsoga Jan 31 '16 at 01:47
Sorry about that! I've updated the answer to include the quotes. – bacongravy Jan 31 '16 at 07:37

bash - hash binary contents of variable without creating a file

2 Answers2