0

C++ examples of MXNet contain model training examples for MNISTIter, MNIST data set (.idx3-ubyte or .idx1-ubyte). However the same code actually recommends to use im2rec tool to produce the data, and it produces the different .rec format. Looks like the .rec format contains images and labels in the same file, because im2rec takes a prepared .lst file with both (number, label and image file name per each line).

I have produced the code like

auto val_iter = MXDataIter("ImageRecordIter");
setDataIter(&val_iter, "Train", vector < string > 
  {"output_train.rec", "output_validate.rec"}, batch_size));

with all files present but it fails because four files are still required in the vector (segmentation fault). But why, should not labels be inside the file now?

Digging more into the code, I found that setDataIter actually sets the parameters. Parameters for ImageRecordIter can be found here. I tried to set parameters like path_imgrec, path.imgrec, then call .CreateDataIter() but all this was not helpful - segmentation fault on the first attempt to use the iterator.

I was not able to find a single example in the whole Internet about how to train any MxNet neural network in C++ using .rec file format for training and validation sets. Is it possible? The only work around I found is to try original MNIST tools that produce files covered by MNIST output examples.

Chaitanya Bapat
  • 3,381
  • 6
  • 34
  • 59
Audrius Meškauskas
  • 20,936
  • 12
  • 75
  • 93

1 Answers1

0

Eventually I have used Mnisten to produce the matching data set so that may input format is now the same as MxNet examples use. Mnisten is a good tool to work, just it is important not to forget that it normalizes grayscale pixels into 0..1 range (no more 0..255).

It is a command line tool but with all C++ code available (and there is not really a lot if it), the converter can also be integrated with existing code of the project to handle various specifics. I have never been affiliated with this project before.

Audrius Meškauskas
  • 20,936
  • 12
  • 75
  • 93