Realm Files in Memory For Production Applications

Question

So, I am little confused about how multiple Realms would work in a single application for a user accessing all of the information in the application.

For example, if Instagram used Realm, am I, as a user, accessing multiple Realms per feature (search, main feed, messages, etc.)? Under current limitations, Realm's fourth stated (current) limitation says

Any single Realm file cannot be larger than the amount of memory your application would be allowed to map in iOS — this changes per device, and depends on how fragmented the memory space is at that point in time (there is a radar open about this issue: (rdar://17119975)). If you need to store more data, you can map it over multiple Realm files.

I take this to mean there is limited space on the device and therefore I either need one Realm per user who may access any and all data or that I still need a backend to store data. Is this correct?

The database part makes sense, but how would map it over multiple Realm files even work?

Answer 1

To understand this limitation requires some knowledge about how operating systems work.

First of all, your iOS device (or desktop computer, or server...) stores data in RAM. Older devices address this memory using 32-bit addresses; newer devices address this memory using 64-bit addresses.

Now, the physical RAM in your computer is all addressed. However, your application does not ever see or work with the addresses of the physical RAM. This is because modern operating systems use virtual memory to provide a layer of indirection, allowing each process to 'pretend' that it is working with its own memory space.

Virtual memory makes it easier for the operating system to implement some interesting features. For example, the mmap system call allows a file on disk to be "mapped" into a process's address space, allowing the process to read and write to that file in the same way as if it were reading to and writing from RAM. Realm uses this feature extensively, and it is part of the reason why Realm is quite performant.

In practice, all devices (whether phones, desktop computers, or servers) have far more possible memory addresses than they have actual RAM. This means that, theoretically, you should be able to mmap a file that is multiple gigabytes large, and still have addresses to spare. (For example, on a 32-bit iOS device, the operating system should be able to provide up to 4 GB of addressable space per process.)

Unfortunately, iOS enforces restrictions on how large a process's virtual memory address space can be, which means that the actual size of Realm files you can open is significantly smaller than the theoretical limit. Exactly how these restrictions work aren't clear to anyone outside Apple, but there are some observations in the linked Radar (which you can read here).

So, the limitation has nothing to do with "Realms per user" or a backend. Rather, it means that iOS won't usually let you open any single Realm file that is more than several hundred megabytes in size.

When we say "map over multiple Realm files", we mean manually "sharding" your data. For example, if you were creating a phone book application and you found that putting all your entries in a single Realm was causing mmap to fail, you might instead split your data up into 3 Realms: one for people whose last names begin with A through H, one for last names beginning with I through S, and one for last names beginning with T through Z.