I would recommend looking at it with a profiler. Then you can see what kind of classes take up space.
A typical pitfall with db4o is that that a 'ObjectContainer' is kept open for a long time, with a high activation depth. Then a large part of your object graph is kept in memory.
Some nobs to try:
configuration.common().weakReferenceCollectionInterval(milli-secs);
How often db4o clears up it's weak reference cache system. If you lower that interval, cleans up more aggressive.
There is a file level cache. I think it's quite low by default. Anyway, here's setting:
Storage fileStorage = new FileStorage();
// A cache with 128 pages of 1024KB size, gives a 128KB cache
Storage cachingStorage = new CachingStorage(fileStorage,128,1024);
configuration.file().storage(cachingStorage);
Maybe there are more caches. I don't remember all of then.