0

Im looking into persisting the content of a Kafka topic in memory in one of my Spring Boot applications.

The compacted Kafka topic contains about 90.000 entries, much like this pseudo code.

//Each entry on the topic will be of type A
A {
    int id1;
    int id2;
    List<B> bs;
    List<C> cs;
}
//A lot of the A's will refer to the same B and C id's but with different toggle values.

B {
    UUID id;
    boolean aToogle;
    .. possibly more
}

C {
    UUID id;
    boolean aToogle;
    .. possibly more
}

The List of A's and B's in the A items will have a variable size from 1 -> 500.000

All the entries should be persisted in a way that supports easy lookup, like a : HashMap<A-Identifier, A>

Doing this as plain POJO's has quite the memory foot print, so i'm wondering if there is a more memory efficient way to persist something like this. Trying to avoid persisting in a database, if at all possible.

itsnotallama
  • 103
  • 1
  • 5
  • 2
    If you want to be able to query the persisted data structure, use a database. If you want a more memory efficient of representing the entire data structure in memory, use a database. (Of some kind. Not necessarily an SQL database ....) AFAIK, all other mechanisms for persisting data will not meet one or both of your criteria. – Stephen C Apr 12 '20 at 14:12
  • You could compress your in-memory objects using gzip: https://stackoverflow.com/questions/5934495/implementing-in-memory-compression-for-objects-in-java – Scarysize Apr 12 '20 at 19:59

0 Answers0