1

I am using a dataset which has 750 sound segments, and I need to load them all in python. I am using librosa for that.

The total size of all the files is 28 GB. When I execute the program to load them, the RAM on my environment (google colab) gets full. I can't load all of them.

I know google colab has the option to add more RAM for money, but is there any hope for me to make the program more efficient for less RAM use?

The segment for reference:

mixture = []
drums = []
bass = []
accompaniment = []
vocals = []
for i in range(150):
  mixture_wav = df.at[i, 'mixture']
  drums_wav = df.at[i, 'drums']
  bass_wav = df.at[i, 'bass']
  accompaniment_wav = df.at[i, 'accompaniment']
  vocals_wav = df.at[i, 'vocals']
  
  y_mixture, sr = lb.load(path + str(i + 1) + '/' +mixture_wav)
  y_drums, sr = lb.load(path + str(i + 1) + '/' +drums_wav)
  y_bass, sr = lb.load(path + str(i + 1) + '/' +bass_wav)
  y_accompaniment, sr = lb.load(path + str(i + 1) + '/' +accompaniment_wav)
  y_vocals, sr = lb.load(path + str(i + 1) + '/' +vocals_wav)
  
  mixture.append(y_mixture)
  drums.append(y_drums)
  bass.append(y_bass)
  accompaniment.append(y_vocals)
  vocals.append(y_vocals)
amoosebitmymom
  • 11
  • 3
  • 1
    If you need to load all the files simultaneously then there is no way to make that take less memory. There is no way to tell from the code you posted if it is possible that you don't need all the information at the same time – Erik McKelvey Dec 08 '21 at 20:04
  • Yeah, more context here would help. Tell us more about what you're trying to do, and you'll probably get suggestions for how to do it without loading all that data at once. – Cody Dec 08 '21 at 20:20
  • Why do you need to load all files in RAM? Can those file be compressed? What are you trying to achieve? – jlandercy Dec 08 '21 at 20:21

0 Answers0