I have lmdb files stored on a nvme ssd to be loaded to memory using the following python code. The server is running ubuntu 18.04. The read speed of the drive (as tested using hdparm) is >17.5G/s (cached) and ~1.6G/s (buffered). 1.6G/s translate to ~4000 images/s in my data. Indeed, the file loading speed starts out with ~3300 image/s for the first lmdb file, which is reasonable considering there is a pickle call, and the data has to be put into a list.
The mysterious thing is, after a while (loading of the first ~6G data), the read speed dropped suddenly to ~700 images/s (or ~350 MB/s) and the loading of the remaining of that file, and following lmdb files stayed at the lower speed. I checked the disk activity using iotop, and the speed drop happened almost exactly at the point when iotop detected there is this fast disk load task.
Therefore, I'm wondering, does Ubuntu automatically limit the IO speed for users? if so, how can I get around it? I can get the sudo privilege if needed. If not, what else could be responsible for this performance drop? Also, is there any apparent way to optimize the code so it could hit the theoretically best performance? (5000 images or 2G per sec) I'm wondering how could I utilize the cached read speed, and I suspect another bottleneck (before the performance drop) could the CPU speed.
self.data_list = []
for i, env in enumerate(self.envs):
with self.envs[i].begin(write=False) as txn:
key = self.keys_list[i]
temp = [pickle.loads(txn.get(key[idx])) for idx in tqdm(
range(self.length_list[i]),
desc="loading from {}".format(self.db_paths[i]),
total=self.length_list[i],
unit=" images",
)]
self.data_list += temp
