1

I have a Golang process that runs SQL queries on a 400MB SQLite file.

I am using https://github.com/mattn/go-sqlite3 with the connection string:

file:mydb.sqlite?mode=ro&_journal=DELETE

When run on my dev machine on Docker it only needs 20MB of RAM, but on Google Run any instance smaller than 512MB will return HTTP code 500 with a memory exceeded limit in the logs.

docker diff x shows that the DB file is not modified (which I assume would cause gVisor to copy the whole binary SQLite db file to RAM to modify it).

How the docker image is built

I am copying the SQLite DB file into the image with the source code:

FROM golang:latest
...
COPY . /go/src/api

I have a global var in my Golang file: var db *sqlx.DB

This gets set in the main fn, before ListenAndServe:

conStr := fmt.Sprintf("file:%s?mode=ro&_journal=DELETE", *fileName)
dbConn, err := sqlx.Open("sqlite3", conStr)
db = dbConn

I query the db within a HTTP request:

err := db.Selectv(&outDataSet, "SELECT...", arg1, arg2)

Why this must be an issue with the Cloud Run environment

docker stats never goes above 20MB when run locally.

Limiting docker run to 20MB RAM also runs fine on my dev machine:

docker run \
  --memory=20m \
  --memory-swap=20m \

The Cloud Run "Container Memory Allocation" metric also stays well below 128M:

https://console.cloud.google.com/monitoring/metrics-explorer

Container Memory Allocation

Thanks.

zino
  • 1,222
  • 2
  • 17
  • 47
  • 4
    I'm not sure that you measure correctly your container memory when it runs locally. You can't load 400Mb file in only 20Mb!! – guillaume blaquiere Jan 19 '20 at 19:10
  • 1
    I am not loading the 400MB file into RAM, I am running SQLite `SELECT` queries on it that return only a fraction of the data. SQLite is disk based storage and I assume it does not load the dataset into memory. – zino Jan 19 '20 at 20:26
  • 1
    If you downvote please leave a comment explaining why. – zino Jan 19 '20 at 20:28
  • @zino, Can you details how do you build your container and share what is done at startup (your go code). I will try to explain you what happen. *Note: I didn't downvote, few ask the question and the others simply set the working memory. Your question is good!* – guillaume blaquiere Jan 19 '20 at 20:37
  • Did you look at the log traces? Any entry with GVisor? It looks like something is forbidden by the platform and thus all is loaded in memory instead of reading directly from "the disk"? – guillaume blaquiere Jan 20 '20 at 06:38
  • Are you writing data to disk? In Cloud Run, there is is no persistent disk. All data you write goes to an overlayfs which lives in memory. – Wietse Venema Jan 23 '20 at 09:20
  • @WietseVenema No I am not, this is why I have set the journal mode of the SQLite DB to `DELETE` and checking with `docker diff x` locally to make sure the db file is not written to. – zino Jan 24 '20 at 18:53
  • @guillaumeblaquiere I cannot read the gVisor logs as its an implementation detail - only Google engineers can see what is happening. – zino Jan 24 '20 at 18:59
  • 1
    My question was: "Is the logs, do you see some warning from gVisor limitation?". If not, it's not the issue! – guillaume blaquiere Jan 24 '20 at 20:56
  • @zino one more thought: did you try setting _query_only? "The query_only pragma prevents all changes to database files when enabled." – Wietse Venema Jan 25 '20 at 21:27
  • @zino Are you able to deploy with < 512 MB of RAM. Just deploy. If so, I suggest you try to lower the concurrency of your service in order to [optimize memory](https://cloud.google.com/run/docs/configuring/memory-limits#optimizing). This will be of use as it is possible that your service runs out of memory when it starts serving a lot of traffic. My suggestion in that case would be to [set concurrency to 1](https://cloud.google.com/run/docs/configuring/concurrency) and see if you still get the `out of memory` errors. – maniSidhu98 Jan 25 '20 at 22:08
  • @WietseVenema The "query_only" pragma will block any write queries from running - I am not sending any write queries to SQLite so this is not an issue. maniSidhu98 Yes I can deploy at 128MB but the service always responds with HTTP 500 - this happens at any concurrency setting. – zino Jan 26 '20 at 15:19
  • Are you logging objects within your setup? This may prevent these objects from being garbage collected. This usually would cause increased file size and consequently, the underlying Infrastructure runs out of memory to handle the setup. If that is the case then you may need to make the logging less verbose. – oakinlaja Jan 26 '20 at 21:24
  • 1
    Here is a suggestion. Before opening the SQLite database, check available memory inside your container. Then open the database. Check memory again. This should give you a clue. In a container, files are stored in read-only layers. When you open a file, a new layer is created. The new layer occupies at least the size of the file that you are opening and is in-memory. For a read-only file this is not really necessary but I cannot see into gVisor to know the implementation details. – John Hanley Jan 28 '20 at 03:25

2 Answers2

4

According to the official documentation:

Configuring Memory Limits

Cloud Run container instances that exceed their allowed memory limit are terminated.

The following count towards the available memory of your container instance:

a.running the application executable (as the executable must be loaded to memory)

b.allocating memory in your application process

c.writing files to the filesystem

The size of the deployed container image does not count towards the available memory.

Also I would suggest to consider:

Are your container instances exceeding memory?

Your container instances might be exceeding the available memory. To determine if this is the case, look for such errors in the varlog/system logs. If the instances are exceeding the available memory, consider increasing the memory limit.

Note that the Cloud Run container instances run in an environment where the files written to the local filesystem count towards the available memory. This also includes any log files that are not written to /var/log/* or /dev/log.

It seems that your container file systems is using the memory.

Community
  • 1
  • 1
marian.vladoi
  • 7,663
  • 1
  • 15
  • 29
  • 1
    I do not think this is the right answer, it was auto selected . The SQLite file is part of the container image, not the runtime file system. Container images do not take runtime RAM. I am not loading the dataset into RAM as proven by limiting RAM to 20MB on my local machine which runs fine. – zino Feb 03 '20 at 13:29
2

In the Cloud Run (fully managed) environment disk storage is an in-memory filesystem. link

pbacterio
  • 1,094
  • 6
  • 12
ahmetlutfu
  • 325
  • 2
  • 9
  • 2
    I am not writing any files to disk at runtime, only reading from the container image using SQLite SELECT queries. – zino Feb 04 '20 at 12:35