One big public Realm that is shared between all users would, in principle, be one way to achieve the result. This has various drawbacks. One of them is, as you mention, that all users would download everything. Another drawback is that there is no privacy or security. Anyone could pretend to be anyone else, and anyone can read anyone else's messages.
A better solution would be to have one Realm per set of users that need to communicate. The Realm would act like a channel between two (or more) users. You would need one big public Realm where all the channels were broadcast. Also you would need to set permissions on the channels.
So, when a user wants to send a message to someone else, they first check if the channel exists. If not, the user creates and distributes permissions for the channel Realm. The user also broadcasts the existence of the channel in the common Realm. All users listen for changes to the common Realm and start sessions with their own channels.