Go 1.13 RSS keeps on increasing, suspected scavenging issue

Question

We are experiencing RSS that keeps on increasing in one of our Go service. We are suspecting it's due to scavenger not returning memory to OS properly (or OS not taking back the memory due to use of MADV_FREE). Checked via pprof, no memory leak detected.

We tried some experiment with the following simple Go program: Go version: go1.13.4 linux/amd64 (tried with go1.13.1 too)

package main
import (
  "fmt"
  "time"
)

func allocate(s int) {
  a := make([]byte, s * 1024 * 1024)
  for i := 0;i < len(a); i += 4096 {
    a[i] = 'x'
  }
  fmt.Printf("%c %d\n", a[0], s)
}

func main() {
  allocate(100)
  allocate(200)
  allocate(500)
  time.Sleep(300 * time.Second)
}

We run with

strace -q -e trace=memory go run main.go

and the results are not consistent:

• https://pastebin.com/sGw3dp9E (no MADV_FREE reported)
• https://pastebin.com/d6CmRfD4 (MADV_FREE reported, but why the values freed are so small?)

running with gctrace, we got: https://pastebin.com/6JaC2r85

we confirmed that RSS remains high via top and pmap:

$ pmap -x 22712
22712:   /tmp/go-build073210395/b001/exe/main
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     576     428       0 r-x-- main
0000000000490000     756     320       0 r---- main
000000000054d000      84      56      32 rw--- main
0000000000562000     120      36      36 rw---   [ anon ]
000000c000000000  983040  819440  397680 rw---   [ anon ]
00007fcdfb2e1000   66684   26780   26780 rw---   [ anon ]
00007fffa3ba2000     132      20      20 rw---   [ stack ]
00007fffa3bdf000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB         1051404  847084  424548

Appreciate if someone could help to clarify the following:

1. According to https://github.com/golang/go/issues/30333, in go 1.13 scavenging should be performed periodically instead of waiting for the 5 minutes interval, but there is no MADV_FREE logged by strace sometimes. gctrace does print logs about scavenging, but seems like it's not really calling MADV_FREE? Am I missing something regarding this scavenging logic in go 1.13?
2. Tried with GODEBUG=madvdontneed=1, result is better, but RSS still hovering at around 500MB, only when we combine madvdontneed with debug.FreeOSMemory() then we got RSS < 30MB. Is this the only way to ensure Go is returning memory back to OS in Linux?

Additional notes: running the same program on windows with go 1.13 seems to have the desired effect, i.e. memory is gradually released back to OS.