Huge persisting memory allocation with go-ethereum `bind.NewTransactor()`?

Question

I am developing a REST service that interacts with a private Ethereum blockhain network. First, I used Java with the Web3j library and Jersey. Everything worked as expected but a single instance of the service (not the geth client) was taking up to 500MB of RAM! As I need several instances (~40) running at the same time for simulation purpose, I wanted something lighter.

Thus I switched to Go (32-bit version, Windows 10) and the original go-ethereum package. However, I was very surprised to find out that the memory consumption of the program rises up to ~250MB as soon as I call and store the result of bind.NewTransactor(), which returns a *TransactOpts. I took a look at the sources but I could not explain such behavior.

Is that normal (and if yes, why?) or am I missing something?

Here is my code:

import (
    "github.com/ethereum/go-ethereum/accounts/abi/bind"
    "strings"
)

const WALLET_PATH string = "path/to/json/wallet"

func main() {

    data, _ := ioutil.ReadFile(WALLET_PATH)
    // up to this point, the program only takes a few MBs of RAM 

    auth, _ := bind.NewTransactor(strings.NewReader(string(data)), "mypassphrase")
    // and here the allocated memory rises up to ~270MB
    _ = auth
}

Duplicate of https://ethereum.stackexchange.com/questions/9833/is-it-possible-to-decrease-ram-usage-of-go-ethereumgeth ? — Ullaakut, Aug 10 '18 at 18:07
@Ullaakut Thanks for the link. However, as I said, I am not talking about the geth node, I am just talking about running the code I posted. Plus, I am not trying to send any transactions so far. Actually, the ethclient connection part of the code above can be removed and still the memory consumption is very high... — Q. Betti, Aug 10 '18 at 18:16
Try [memory profiling](https://blog.golang.org/profiling-go-programs) it. — Michael Hampton, Aug 10 '18 at 19:40
Geth is a fairly large program, and the internal code is quite interlinked. Using even a subset of geth can require running a lot of code. — Raghav Sood, Aug 10 '18 at 20:08
@RaghavSood Indeed. Please have a look at my edited question and my answer, you'll see that I am only dealing with cryptographic implementation issues so I do not think that Geth's size is involved howsoever. — Q. Betti, Aug 15 '18 at 20:00

Q. Betti · Answer 1 · 2018-08-15T20:31:42.703

Ok so, I did not really found the answer to my question, but as I did find a workaround I'll share it anyway.

Memory profiling

As @MichaelHampton suggested, I memory profiled my program. top 10 command revealed the following:

(pprof) top 10
Showing nodes accounting for 256MB, 99.88% of 256.30MB total
Dropped 15 nodes (cum <= 1.28MB)
    flat  flat%   sum%        cum   cum%
    256MB 99.88% 99.88%      256MB 99.88%  github.com/ethereum/go-ethereum/vendor/golang.org/x/crypto/scrypt.Key
        0     0% 99.88%      256MB 99.88%  github.com/ethereum/go-ethereum/accounts/abi/bind.NewTransactor
        0     0% 99.88%      256MB 99.88%  github.com/ethereum/go-ethereum/accounts/keystore.DecryptKey
        0     0% 99.88%      256MB 99.88%  github.com/ethereum/go-ethereum/accounts/keystore.decryptKeyV3
        0     0% 99.88%      256MB 99.88%  github.com/ethereum/go-ethereum/accounts/keystore.getKDFKey
        0     0% 99.88%   256.01MB 99.88%  main.main
        0     0% 99.88%   256.30MB   100%  runtime.main

As you can see, the memory consumption comes from the Key function in package scrypt, which is called indirectly by bind.NewTransactor(...). From the doc:

Key derives a key from the password, salt, and cost parameters, returning a byte slice of length keyLen that can be used as cryptographic key.

In particular, this corresponds to how a private key is generated from a JSON wallet file. Indeed, the function itself allocates a lot of memory for cryptographic computations. What I do not understand though is why this huge memory allocation seems to persist once the key has been generated (the persistence is observed when I start the server with log.Fatal(http.ListenAndServe(":8080", nil)) just after).

Workaround

To confirm my suspicion, I used another way to generate my *TransactOpts, which consists in getting the key directly form its hexadecimal representation instead of generating it from a wallet file:

func main() {
    privateKey, err := crypto.HexToECDSA("myKeyInHex")

    if err != nil {
        log.Fatal(err)
    }

    auth := bind.NewKeyedTransactor(privateKey)

    _ = auth
}

Although bind.NewTransactor and bind.NewKeyedTransactor return exactly the same object (the only difference being how the key was generated), using bind.NewTransactor results in a 256MB persisting memory allocation, instead of a few KB for bind.NewKeyedTransactor as shown below:

(pprof) top 10
Showing nodes accounting for 11.04kB, 100% of 11.04kB total
Showing top 10 nodes out of 19
    flat  flat%   sum%        cum   cum%
  6.83kB 61.90% 61.90%     6.83kB 61.90%  time.initLocalFromTZI
  4.21kB 38.10%   100%     4.21kB 38.10%  github.com/ethereum/go-ethereum/crypto/sha3.(*state).clone (inline)
       0     0%   100%     4.21kB 38.10%  github.com/ethereum/go-ethereum/accounts/abi/bind.NewKeyedTransactor
       0     0%   100%     4.21kB 38.10%  github.com/ethereum/go-ethereum/crypto.Keccak256
       0     0%   100%     4.21kB 38.10%  github.com/ethereum/go-ethereum/crypto.PubkeyToAddress
       0     0%   100%     4.21kB 38.10%  github.com/ethereum/go-ethereum/crypto/sha3.(*state).Sum
       0     0%   100%     6.83kB 61.90%  github.com/pkg/profile.Start
       0     0%   100%     6.83kB 61.90%  github.com/pkg/profile.Start.func2
       0     0%   100%     6.83kB 61.90%  log.(*Logger).Output
       0     0%   100%     6.83kB 61.90%  log.(*Logger).formatHeader

Thus, for my simulation I will generate my private keys from my JSON wallets and store them in text files upstream, and then use bind.NewKeyedTransactor(...). I know this is not safe howsoever but for my simulation purpose it will be sufficient.

However, I'm pretty sure bind.NewTransactor does not have the expected behavior concerning memory consumption, so I am going to open an issue on the go-ethereum repository.

Huge persisting memory allocation with go-ethereum `bind.NewTransactor()`?

1 Answers1

Memory profiling

Workaround