1

Every time I try to execute a transaction or query where the payload is > ~2MB I get the following errors:

Immediately upon executing the query, from the docker container running the business network application:

[ERROR] lib/handler.js - Chat stream with peer - on error: 
"Error: 8 RESOURCE_EXHAUSTED: Received message larger than max (19090846 vs. 4194304)\n
at createStatusError (/usr/local/src/node_modules/grpc/src/client.js:64:15)\n
at ClientDuplexStream._emitStatusIfDone (/usr/local/src/node_modules/grpc/src/client.js:270:19)\n
at ClientDuplexStream._receiveStatus (/usr/local/src/node_modules/grpc/src/client.js:248:8)\n
at /usr/local/src/node_modules/grpc/src/client.js:804:12"

Then from the application side, when the timeout has been reached:

{ Error: 2 UNKNOWN: error executing chaincode: failed to execute transaction: timeout expired while executing transaction
at new createStatusError (C:\Users\jean5\Fabric\Qostodian\qostodian-analyzer\node_modules\grpc\src\client.js:64:15)
at C:\Users\jean5\Fabric\Qostodian\qostodian-analyzer\node_modules\grpc\src\client.js:583:15
code: 2,
metadata: Metadata { _internal_repr: {} },
details: 'error executing chaincode: failed to execute transaction: timeout expired while executing transaction' }

These errors show that the GRPC default limit of 4MB is reached when I try to retrieve ~18.2MB of data from the query (19090846 vs. 4194304).

From what I've seen Fabric is hardcoded to support up to 100MB already:

MaxRecvMsgSize = 100 * 1024 * 1024
MaxSendMsgSize = 100 * 1024 * 1024

I've also found a JIRA task (FAB-5049) on hyperledger.org where they face the same issue. However, there is no discussion of a potential fix for the 4MB limit.

Question 1: If fabric is hardcoded with 100MB, where is that 4MB limit coming from?

Question 2: How can I make sure that the GRPC limit is indeed 100MB?

I would also like to know if its possible to explicitly set the GRPC limit for example in the connection.json or when installing/starting the network using composer CLI.

Jean
  • 47
  • 7

1 Answers1

3

GRPC imposes a default limit of 4Mb. When a connection over grpc is set up you can specify an alternative limit. To do this with Composer 0.19 you need to modify the connection profile which defined the hyperledger fabric you connect to and provide gprcOptions which contains options recognised by the node implementation of grpc. An example of setting these options on a peer is given here.

"peer0.org1.example.com": {
    "url": "grpcs://peer0.org1.example.com:7051",
    "eventUrl": "grpcs://peer0.org1.example.com:7053"
    "grpcOptions": {
        "ssl-target-name-override": "peer.org1.example.com",
        "grpc.keepalive_time_ms": 600000,
        "grpc.max_send_message_length": 15728640,
        "grpc.max_receive_message_length": 15728640
    },
    "tlsCACerts": {
        "pem": "-----BEGIN CERTIFICATE----- <etc> "
    }
}

you can also set the grpcOptions for an orderer as well in a similar manner. Note the message lengths are number of bytes, a value of -1 means unlimited.

Unfortunately there is no way to update existing cards in the card store with new profiles at the moment via the cli. If you are using a file system card store then you can replace the connection.json file manually for your cards.

david_k
  • 5,843
  • 2
  • 9
  • 16
  • I've looked at the doc and it seems like it is grpc-max-send-message-length (see lhttps://hyperledger.github.io/composer/next//reference/connectionprofile) Are you sure it really is 'grpc.' instead of 'grpc-' ? – guiomie Apr 04 '18 at 01:35
  • I've tried all variations of the grpcOptions including with "grpc.", "grpc-", and without "grpc" for the `max-send-message-length` and `max-send-message-length` parameters. I deleted the entire BNA, restaged the network, make sure that all connection.json files contains the right config and I still hit the 4MB limit. – Jean Apr 04 '18 at 01:39
  • @guiomie the doc will be updated to specify 'grpc.' as that appears to be the correct value as defined by the fabric node sdk code implementation, however looking at the fabric node sdk in more detail their defaults are set to -1 which means this should have been unlimited anyway. Either a change to grpc has altered these options or something else is imposing a limit now – david_k Apr 04 '18 at 07:15
  • Just noticed a mistake in the above example, it should have been grpc.max_receive_message_length and grpc.max_send_message_length (underscores, not dashes) – david_k Apr 04 '18 at 07:29
  • try using "_" instead of "-", e.g. "grpc.max_receive_message_length" – Gari Singh Apr 04 '18 at 11:12
  • I've tried the "_" (i.e., grpc.max_receive_message_length) to no avail. I'm starting to wonder if the limitation is imposed somewhere else as @david_k points out. Could it be from the Fabric ccenv image when we start the network for the first time and the docker container is configured? Could there be a required config in the docker-compose.yml? – Jean Apr 04 '18 at 16:35
  • It looks like it is a limit between the chaincode container and the peer, I've managed to hit the 100Mb limit by trying to create lots of very large assets in a txn, but not trying to pass a large amount of data in. I've not hit it either trying to retrieve all the assets back which I must admit I was expecting to exceed to 100Mb limit with the number I have created. Is the problem always when you send in a large payload or does it also happen when returning a large payload ? only ask as the error at the top indicates inbound payload – david_k Apr 04 '18 at 19:46
  • I've gotten the same 4MB error for both inbound and outbound transactions. For example, try to create an asset with a string of text totaling >4MB (we take the content of a local .txt file). When you commit or retrieve that asset the error will occur. The same happens if we retrieve many smaller assets where the sum of their content > 4MB. – Jean Apr 04 '18 at 21:40
  • I am really thinking this is an issue with the node chaincode container support in fabric not setting the 100Mb defaults on node.js chaincode containers and so you get the default 4Mb limit. – david_k Apr 04 '18 at 21:58
  • ok, I've proven this is an issue with node.js chaincode support in fabric and have raised an issue https://jira.hyperledger.org/browse/FAB-9378 – david_k Apr 05 '18 at 15:34
  • @david_k Where/how can we get a docker image with the changes you've included? – Jean Apr 16 '18 at 19:46
  • @Jean although the fix is in the fabric codebase now, composer needs to wait for a release of fabric before we can include it in a release. – david_k Apr 20 '18 at 06:33