Deep-learning for mapping large binary input

Question

this question may come as being too broad, but I will try to make every sub-topic to be as specific as possible.

My setting: Large binary input (2-4 KB per sample) (no images) Large binary output of the same size

My target: Using Deep Learning to find a mapping function from my binary input to the binary output.

I have already generated a large training set (> 1'000'000 samples), and can easily generate more.

In my (admittedly limited) knowledge of Neural networks and deep learning, my plan was to build a network with 2000 or 4000 input nodes, the same number of output nodes and try different amounts of hidden layers. Then train the network on my data set (waiting several weeks if necessary), and checking whether there is a correlation between in- and output.

Would it be better to input my binary data as single bits into the net, or as larger entities (like 16 bits at a time, etc)?

For bit-by-bit input: I have tried "Neural Designer", but the software crashes when I try to load my data set (even on small ones with 6 rows), and I had to edit the project save files to set Input and Target properties. And then it crashes again.

I have tried OpenNN, but it tries to allocate a matrix of size (hidden_layers * input nodes) ^ 2, which, of course, fails (sorry, no 117GB of RAM available).

Is there a suitable open-source framework available for this kind of binary mapping function regression? Do I have to implement my own?

Is Deep learning the right approach?

Has anyone experience with these kind of tasks? Sadly, I could not find any papers on deep learning + binary mapping.

I will gladly add further information, if requested.

Thank you for providing guidance to a noob.

Answer 1

You have a dataset containing pairs of binary valued vectors, with a max length of 4,000 bits. You want to create a mapping function between the pairs. On the surface, that doesn't seem unreasonable - imagine a 64x64 image with binary pixels – this only contains 4,096 bits of data and is well within the reach of modern neural networks.

As your dealing with binary values, then a multi-layered Restricted Boltzmann Machine would seem like a good choice. How many layers you add to the network really depends on the level of abstraction in the data.

You don’t mention the source of the data, but I assume you expect there to be a decent correlation. Assuming the location of each bit is arbitrary and is independent of its near neighbours, I would rule out a convolutional neural network.

A good open source framework to experiment with is Torch - a scientific computing framework with wide support for machine learning algorithms. It has the added benefit of utilising your GPU to speed up processing thanks to its CUDA implementation. This would hopefully avoid you waiting several weeks for a result.

If you provide more background, then maybe we can home in on a solution…