Why does Varnish eat up all memory despite having both transient and malloc limits?

Question

I'm using Varnish to cache static files in front of an Apache2 webserver. It's a mix of small files (CSS/JS/images) and large files (from 200 MB to 2 GB).

I've limited both transient and malloc storage in the daemon to 8/12 GB respectively:

ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :8000,PROXY -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,12G -s Transient=malloc,8G

This seems to be respected when checking varnishstat:

SMA.s0.g_bytes           2.67G
SMA.s0.g_space           9.33G
                        12.00G (total)
SMA.Transient.g_bytes    3.26G
SMA.Transient.g_space    4.74G
                         8.00G (total)

But not so when checking top:

MiB Mem :  32169.8 total,    274.4 free,  30405.1 used,   1490.3 buff/cache
MiB Swap:  16382.0 total,   9683.5 free,   6698.5 used.   1360.4 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 3118 vcache    20   0   51.4g  29.3g   1.3m S   7.9  93.2   3:30.05 /usr/sbin/varnishd

This memory usage causes the server to swap, then thrash and eventually invoke oom-killer to destroy the Varnish daemon.

The server is a VM with 32 GB memory. I've also tried with other transient/malloc limits and piping the large files with no difference in behavior.

What could be the issue?

VCL:

backend default {
    .host = "127.0.0.1";
    .port = "8001";
}

sub vcl_recv {
    unset req.http.X-Cache-Request;

    // Unset cookies and indicate that this request should be cached
    if (req.http.host ~ "^static.example.com(:\d+)?$"){
        unset req.http.Cookie;
        set req.http.X-Cache-Request = "1";
    }
}

sub vcl_backend_response {
    // Set TTL if request that should be cached and status code is 2xx, 3xx or 4xx
    if (
        bereq.http.X-Cache-Request
        && beresp.status >= 200 && beresp.status < 500
        && beresp.http.Cache-Control !~ "no-cache"
    ){
        unset beresp.http.Set-Cookie;
        set beresp.ttl = 1w;
    }
}

sub vcl_deliver {
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    } else {
        set resp.http.X-Cache = "MISS";
    }
}