I have a data set containing 3-dimensional labeled numerical (float) data stored in two directories named, say, NEGATIVE and POSITIVE, respectively. In each directory, each data point (a vector actually) is written in an individual .dat (or .txt) file. Each original data vector is 3D meaning that it has a shape of, for example, 646464, which is flattened into a 1D array before written in the file (the file then contains one column and 64*64*64 rows).
I want to build a neural network in Python with TensorFlow (Keras) taking these .dat files as inputs (just like taking images but now numerical data instead). However, there doesn't seem to be a function facilitating us to do that. I thought about loading the data files all together and manually reshape them to a training list and a test list containing numpy arrays (each element of the list has shape 64*64*64 -> a single data vector) and also lists of their labels. , but that would require insane amount of RAM (data set is really large). So I wonder if there is a way that I can load them in the form of small batches for the network to train (used in the .fit() method), just like one would do when loading images by 'image_dataset_from_directory'?
