I have a 2D array saved in txt file(10 millions rows). Because it's too large, I need to load it by chunk. Let's say read every 1000 lines each time (as a batch size of training data in Neural Network). Now I followed this :
Read specific lines from text file as numpy array
It works well. But it is way too slow. Is there any other way to do this please?
from itertools import islice
import numpy as np
data=np.ones((10000000,100))
# This is sample data
#I saved data using
outfile= ('data.txt','wb')
np.savetxt(outfile, data)
#Now load data
file = open('data.txt','rb')
array = np.genfromtxt(islice(file, 1000000,1000005))
Or is there any other way to save and load data by chunks faster?
