How to create SQLite in-memory database at specific address

Question

I want to use SQLite to store some metadata information within a file. This file is already mmap'd. What I want to be able to do is create a SQLite DB, passing in a char* to tell it where to make the DB, instead of having it allocate its own. Can this be done? SQLite docs for in-memory databases just say to use ":memory:", and that the DB will be destroyed at end of process, with no indication of how to use already-existing data or persist it.

If not, what workarounds are there on Linux? Is there an "inverse" mmap, so I can take an address and map it to a new file? (So /foo would be a view onto a part of /bar?)

score 5 · Answer 1 · answered May 07 '18 at 06:42

A VFS is definitely the way to go, but you don't even need to write your own: The sqlite3 distribution contains a file ext/misc/memvfs.c that already implements what you need:

USAGE:

sqlite3_open_v2("file:/whatever?ptr=0xf05538&sz=14336&max=65536", &db,
                   SQLITE_OPEN_READWRITE | SQLITE_OPEN_URI,
                   "memvfs");

These are the query parameters:

   ptr=          The address of the memory buffer that holds the database.

   sz=           The current size the database file

   maxsz=        The maximum size of the database.  In other words, the
                 amount of space allocated for the ptr= buffer.

   freeonclose=  If true, then sqlite3_free() is called on the ptr=
                 value when the connection closes.

The ptr= and sz= query parameters are required.  If maxsz= is omitted,
then it defaults to the sz= value.  Parameter values can be in either
decimal or hexadecimal.  The filename in the URI is ignored.

score 1 · Accepted Answer · answered Apr 21 '16 at 12:30

1

In-memory databases are just databases whose page cache is not saved to disk.

There is no mechanism to access these pages.

Your only choice would be to implement your own VFS to redirect file accesses to the correct part of that file.

answered Apr 21 '16 at 12:30

CL.

173,858
17
217
259

"In-memory databases are just databases whose page cache is not saved to disk." That must be in reference specifically to SQLite in-memory databases, because it is definitely not true of in-memory database systems in general. – Steven Graves Apr 21 '16 at 22:57

score 1 · Answer 3 · answered Apr 21 '16 at 23:02

1

I'm not a SQLite expert, but it seems like you could use the information here to use your own allocator, and control the memory location at which SQLite creates the in-memory database: https://www.sqlite.org/c3ref/mem_methods.html

answered Apr 21 '16 at 23:02

Steven Graves

837
6
9

But you don't know what allocation is used for what purpose. – CL. Apr 22 '16 at 07:06
Does it matter? If your objective is control what memory SQLite uses (i.e. know/control the memory address at which the IMDB is created), it seems like this does it. Maybe I'm missing something... – Steven Graves Apr 22 '16 at 17:23
I don't even mind creating it on-disk. But I need to open it from memory. Looks like a VFS is the way to go. – MichaelGG Apr 23 '16 at 02:11

How to create SQLite in-memory database at specific address

3 Answers3