How to read from a binary file and load the data into a 3d vector in C++?

Question

I currently have some .bin files, each of which contains a matrix of size AxBxC. I would like to load it into a 3d vector vector<vector<vector<float>>> in C++, but encountered some problems.

I tried to flatten the matrix and load it into a buffer first, but the size of the buffer was weird so I got stuck there.

My code was:

vector<vector<vector<float>>> load_3d_bin(const std::string& path){
    vector<vector<vector<float>>> r;
    std::ifstream binary_feat(path.c_str(),std::ios::in | std::ios::binary);
    std::vector<float> flattened_feat((std::istream_iterator<char>(binary_feat)),std::istream_iterator<char>());
    for(auto i: flattened_feat){
        int value = i;
    }
    std::cout<<flattened_feat.size()<<endl;

// code to be written

   return r;
}

Ideally r would be a 3d vector, but I am stuck at the iterator part as the .size() outputs differently from total length = AxBxC, which was what I expected...

Any solution or feedback? Thanks in advance!

--Edit:

Sorry for not being clear enough! In my original .bin files, the floats are encoded as 3d matrices, i.e., in each file they are in an array of AxBxC size. I would like to first flatten the original 3d matrices and then reshape the flattened vectors into 3d vectors of size 'AxBxC'.

I managed to do the flattening part myself as follows:

typedef vector<vector<vector<float>>> t_f
typedef vector<vector<float>> m_f

t_f load_3d_bin(const std::string& path){
    t_f r;
    std::ifstream binary_feat(path.c_str(), std::ios::in | std::ios::binary);
    std::vector<float> flattened_feat;
    float tmp = 0;
    binary_feat.read(reinterpret_cast<char*>(&tmp), sizeof(float));
    while(!binary_feat.eof()) {
        binary_feat.read(reinterpret_cast<char*>(&tmp), sizeof(float));
        flattened_feat.push_back(tmp); 
    }
    std::cout<<"file size:::"<<flattened_feat.size()<< std::endl;

# the reshaping code to be written

return r;
}

And now with the flattened vector I have, I tried to reshape it into a 3d vector, which is the # the reshaping code to be written part above:

t_f r(A, m_f(B));
for (int i=0; i<A; i++){
    for (int j=0; j<B; j++){
        for (int k=0; k<C; k++){
            r[i][j][k] = flattened_feat[(i*B+j)*C+k];
        }
    }
}

But I got segmentation fault... Any idea why I got the error and how to solve it? Or is it better to directly load the 3d matrices in .bin files to 3d vectors? Thanks!

3d arrays (array in the general sense) can be realized by using a 1d array (e.g. std::vector) and managing the 3d strides and indices yourself. This is IMHO a better way than a "real" 3d data structure. It is the method used by opencv, numpy and others. And in addition it makes reading a file rather trivial. — wohlstad, Sep 23 '22 at 09:44
@TedLyngmo The original 3d `.bin` files were written in python... simply `np.array(3dfeatures).astype(np.float32).tofile(3dbin_path)`, where `3dfeatures` is a 3d np array. — Ziggy1209, Sep 26 '22 at 01:44
@Ziggy1209 Perhaps you should include the shortest possible python script that produces such a file in the question. It could give the those interested something to test on when trying to deserialize it in a different language. — Ted Lyngmo, Sep 26 '22 at 16:15

score 0 · Answer 1 · answered Sep 23 '22 at 10:01

Assuming your bin files just contain a flat concatenation of floats in binary format, in row-major order:

vector<vector<vector<float>>> load_3d_bin(const std::string& path){
  vector<vector<vector<float>>> ret(A, vector<vector<float>>(B));
  std::ifstream binary_feat(path.c_str(),std::ios::in | std::ios::binary);

  for (int i = 0; i < A; i++) {
    for (int j = 0; j < B; j++) {
      for (int k = 0; k < C; k++) {
        auto& row = ret[i][j].emplace_back(C);
        binary_feat.read(row.data(), row.size() * sizeof(float));
      }
    }
  }
  return ret;
}

Alternatively, you could use the fantastic armadillo library:

fcube c;
c.load(path.c_str(), raw_binary);
c.reshape(A, B, C);

score 0 · Answer 2 · answered Sep 26 '22 at 04:26

In the end I managed to do this myself.

The solution is as follows:

typedef vector<vector<vector<float>>> t_f
typedef vector<vector<float>> m_f
typedef vector<float> v_f

t_f load_3d_bin(const std::string& path){
    std::ifstream binary_feat(path.c_str(), std::ios::in | std::ios::binary);
    std::vector<float> flattened_feat;
    float tmp = 0;
    binary_feat.read(reinterpret_cast<char*>(&tmp), sizeof(float));
    while(!binary_feat.eof()) {
        binary_feat.read(reinterpret_cast<char*>(&tmp), sizeof(float));
        flattened_feat.push_back(tmp); 
    }

    t_f r(A, m_f(B,v_f(C)));
    for (int i=0; i<A; i++){
        for (int j=0; j<B; j++){
            for (int k=0; k<C; k++){
                r[i][j][k] = flattened_feat[(i*B+j)*C+k];
            }
        }
    }

    return r;
}

The returned r is a 3d vector<vector<vector<float>>> of size AxBxC.

`while(!binary_feat.eof())` is wrong. You'll likely read one `float` too many that way. Why do you read and discard a `float` at the start? Just do `while(binary_feat.read(buf, sizeof(float))) { ... }`. It risky reading directly into the `float`. I'd recommend reading into a `char buf[sizezof(float)]` then `memcpy` the `buf` into the `float`. — Ted Lyngmo, Sep 28 '22 at 21:34

How to read from a binary file and load the data into a 3d vector in C++?

2 Answers2