What's the quickest way to bit shift and bit mask every number in a file using bash, awk or sed?

Question

I have a file (tmp1) which is a list of numbers which are in the following format:

Each 64 bit number here is actually made up of multiple 16 bit fields which have the numbers I'm interested in.

For Eg (and only showing the 48 bits that matter here):

4435738625 = 0000000000000001 0000100001100100 0000000000000001

And the numbers I want are:

a= 0000000000000001 = 1
b= 0000100001100100 = 2148
c= 0000000000000001 = 1

This is the code I'm using to do this right now - but it's painfully slow. The input file contains between 500K and 1 million lines, so I'm trying to look for ways to do this faster or more efficiently.

while read line; do
  a=$((((line >> 32)) & 65535));
  b=$((((line >> 16)) & 65535));
  c=$((line & 65535));
  printf "$a $b $c\n" >>tmp2
done <tmp1

I need to run this on a FreeBSD machine - so I can't use gawk. And awk does not seem to allow bit-wise operations.

Try this `awk` script : http://www.cs.ucsb.edu/~sherwood/awk/decimal2binary.awk.txt — Håkon Hægland, Oct 16 '13 at 13:38
see this : http://stackoverflow.com/questions/5488551/how-to-unpack-64-bit-unsigned-long-in-64-bit-perl — michael501, Oct 16 '13 at 17:49

score 3 · Answer 1 · answered Oct 16 '13 at 13:53

There is a solution, works for your needs. but if it works faster than yours, I cannot tell. you can test.

here I just test with one number in your example, you can wrap it in a loop.

kent$  printf "%064s\n" "$(bc <<< "obase=2;4435738625")"|sed -r 's/.{16}/ibase=2;&\n/g'|bc
1
2148
1

rici · Accepted Answer · 2013-10-16T16:15:01.013

You don't really need bit operations to do that. For example:

awk '{val = $1;
      c = val%65536; val = (val-c)/65536;
      b = val%65536; val = (val-b)/65536;
      a = val%65536;
      print a, b, c}'

However, awk values are not 64-bit integers; they are doubles, which only have 53 bits of precision. So that will only work if none of your numbers are greater than 9007199254740992 (2⁵³).

You might consider using a different tool, like bc or python. If you have GNU bc available (which is possible, even on a BSD system), the following very similar program should work:

bc <(echo 'define s(val) {
             c = val%65536; val = (val-c)/65536
             b = val%65536; val = (val-b)/65536
             a = val%65536;
             print a," ",b," ",c,"\n";
           }
           while(1){v=read(); if (v==0) break; v=s(v);}') \
   < datafile

Note well: the bc read function does not check for end of file, so you need to put some kind of explicit terminator in. I used 0 in the above script, but that might be a valid input in your case. You might want to change it to -1 or some other special value. Whatever, make sure that your datafile is actually terminated with that value.

This works really well, expect for the value of a. Instead of 1 I get 0 because I'm in effect doing (65536%65536) — egorulz, Oct 17 '13 at 05:32
@egorulz: Both the awk and the bc programs work fine for me with your sample inputs. — rici, Oct 17 '13 at 05:38

mr.spuratic · Answer 3 · 2013-10-17T15:03:18.487

Use bc with obase set, and its documented behaviour:

For bases greater than 16, bc uses a multi-character digit method of printing the numbers where each higher base digit is printed as a base 10 number. The multi-character digits are separated by spaces.

$ bc -q <(echo "obase=65536") tmp1 <(echo "halt")
00001 01200 00001
00001 01852 00001
00001 02007 00001
00001 02148 00001

and pipe that into awk if you need to finesse the output a little, e.g. drop the leading zeroes, or deal with variable number of columns (4 columns if ≥ 2⁴⁸, 3 if ≥ 2³² etc.):

| nawk '{printf("%i %i %i\n",(NF>2)?$(NF-2):0,(NF>1)?$(NF-1):0,$NF)}'

The <(echo ...) parts allow bc to read the echo output as a file, a quick alternative to adding those lines to the top and bottom of every input file.

What's the quickest way to bit shift and bit mask every number in a file using bash, awk or sed?

3 Answers3