0

Does someone know a good way to create a pool of objects containing really heavyweight objects?

I already tried it and it works so far, but my problem is, that each object needs to load the same huge model file for initialization (~200MB), so the initialization of the pool takes around two and a half minutes and takes a lot of memory.

Currently, I have an array of objects that will be filled with those objects by a for loop and I know about classes like GenericObjectPool, but I don't think it would make a difference or am I wrong?

Draken
  • 3,134
  • 13
  • 34
  • 54
bunzJ
  • 3
  • 2
  • Why would each object load the same file? – Kayaman Apr 07 '17 at 10:49
  • How do you load the model? Why do you load it multiple times, when it's such a heavy operation? – SME_Dev Apr 07 '17 at 10:49
  • Could the objects share some data? Or can you avoid loading the whole object, i.e. store some in a filoe. 200Mb is a lot for an object. Maybe you should post the code for these objects. – Steve Smith Apr 07 '17 at 10:58
  • My pool saves tokenizer as objects. Depending on the language a pool of tokenizers will be created. E.g. tokenizer for English, spanish etc. The problem is, that it is working without any problem for most languages since tokenizers for English etc. aren't that "heavy", but I'm trying to implement a Chinese tokenizer, which also uses this pool of tokenizers. Unlike the others, the Chinese tokenizer takes a CRF model for tokenization, which has to be loaded from a file. So when I create 10 tokenizer objects, each object has to load the same model file. – bunzJ Apr 07 '17 at 11:59

1 Answers1

0

You can use object pooling API ( i;e Apache API) May be it resolve your problem.

Muhammad Iqbal
  • 144
  • 1
  • 1
  • 12