3

I need to append many binary files in one binary file. All my binary files are saved i one folder:

   file1.bin
   file2.bin
   ...

For that I try by using this code:

import numpy as np
import glob
import os
Power_Result_File_Path ="/home/Deep_Learning_Based_Attack/Test.bin"
Folder_path =r'/home/Deep_Learning_Based_Attack/Test_Folder/'  
os.chdir(Folder_path)
npfiles= glob.glob("*.bin")
loadedFiles = [np.load(bf) for bf in binfiles]
PowerArray=np.concatenate(loadedFiles, axis=0)
np.save(Power_Result_File_Path, PowerArray)

It gives me this error:

    "Failed to interpret file %s as a pickle" % repr(file))
OSError: Failed to interpret file 'file.bin' as a pickle

My problem is how to concatenate binary file it is not about anaylysing every file indenpendently.

tierrytestu
  • 119
  • 4
  • 12
  • 2
    @Bellian , In that case they loading a text file not a binary file, in my case It is absolutely that numpy library is not able to interpret a binary file for that I have that pickle error. – tierrytestu Aug 03 '17 at 14:17
  • The answers given in the linked questions will not help to solve this question. Not a duplicate in my opinion. – Eskapp Aug 03 '17 at 14:23
  • Hmm good point.. is the `binfiles` a typo or where is it from? I only see definition of `npfiles` – Bellian Aug 03 '17 at 14:28

2 Answers2

0

Taking your question literally: Brute raw data concatenation

files = ['my_file1', 'my_file2']
out_data = b''
for fn in files:
  with open(fn, 'rb') as fp:
    out_data += fp.read()
with open('the_concatenation_of_all', 'wb') as fp:
  fp.write(out_data)

Comment about your example

You seem to be interpreting the files as saved numpy arrays (i.e. saved via np.save()). The error, however, tells me that you didn't save those files via numpy (because it fails decoding them). Numpy uses pickle to save and load, so if you try to open a random non-pickle file with np.load the call will throw an error.

GPhilo
  • 18,519
  • 9
  • 63
  • 89
  • I have a binary file, file.bin – tierrytestu Aug 03 '17 at 14:34
  • that makes no difference. When you open the file as 'rb' it reads it as a bytes string (which is also the type of `out_data`, note the `b`before the empty string. [**that `b`is very important**](https://stackoverflow.com/questions/6224052/what-is-the-difference-between-a-string-and-a-byte-string)) – GPhilo Aug 03 '17 at 14:35
0
for file in files:
  async with aiofiles.open(file, mode='rb') as f:
    contents = await f.read()
  if file == files[0]:
    write_mode = 'wb'  # overwrite file
  else:
    write_mode = 'ab'  # append to end of file

  async with aiofiles.open(output_file), write_mode) as f:
    await f.write(contents)
Max Gómez
  • 39
  • 5