I created a transaction processor that stores a 5MB payload. As it does (a 500 transaction storage loop), memory consumption increases until there is no more RAM space (about 220 transactions are submitted) on my 32GB RAM server. At this point, a validator stops working with the following error message:
influxdb_1 | [httpd] 172.22.0.5 - - [21/Dec/2019:19:02:52 +0000] "POST /write?db=metrics&precision=s HTTP/1.1" 204 0 "-" "Python-urllib/3.6" 7d4dc556-2424-11ea-8377-0242ac16000e 112925
sawtooth-validator-default-1 | [2019-12-21 19:02:54.397 ERROR threadpool] (Consensus) Unhandled exception during execution of task _HandlerManager.execute.<locals>.wrapped
sawtooth-validator-default-1 | Traceback (most recent call last):
sawtooth-validator-default-1 | File "/usr/lib/python3/dist-packages/sawtooth_validator/concurrent/threadpool.py", line 83, in wrapper
sawtooth-validator-default-1 | return_value = fn(*args, **kwargs)
sawtooth-validator-default-1 | File "/usr/lib/python3/dist-packages/sawtooth_validator/networking/dispatch.py", line 444, in wrapped
sawtooth-validator-default-1 | return callback(self._handler.handle(connection_id, message))
sawtooth-validator-default-1 | File "/usr/lib/python3/dist-packages/sawtooth_validator/consensus/handlers.py", line 378, in handle
sawtooth-validator-default-1 | self._proxy.validate_block(block_id)
sawtooth-validator-default-1 | File "/usr/lib/python3/dist-packages/sawtooth_validator/consensus/proxy.py", line 225, in validate_block
sawtooth-validator-default-1 | block = next(self._block_manager.get([block_id]))
sawtooth-validator-default-1 | File "/usr/lib/python3/dist-packages/sawtooth_validator/ffi.py", line 203, in __next__
sawtooth-validator-default-1 | payload = from_rust_vec(vec_ptr, vec_len, vec_cap)
sawtooth-validator-default-1 | File "/usr/lib/python3/dist-packages/sawtooth_validator/ffi.py", line 106, in from_rust_vec
sawtooth-validator-default-1 | py_bytes = bytes(vec_ptr[:vec_len.value])
sawtooth-validator-default-1 | MemoryError
influxdb_1 | [httpd] 172.22.0.18 - - [21/Dec/2019:19:02:54 +0000] "POST /write?db=metrics&precision=s HTTP/1.1" 204 0 "-" "Python-urllib/3.6" 7e939083-2424-11ea-8379-0242ac16000e 25652
sawtooth-validator-default-1 | [2019-12-21 19:02:56.425 INFO (unknown file)] [src/journal/block_validator.rs: 265] Block 4a6a1342c829d81752127f5b95deaff1b9cac847ec3675714a2b1b086d281e6f44979beef8506041a89d841e089d8ae36e2513e3f65a91c2e634fa23a4052e06 passed validation
influxdb_1 | [httpd] 172.22.0.4 - - [21/Dec/2019:19:02:58 +0000] "POST /write?db=metrics&precision=s HTTP/1.1" 204 0 "-" "Python-urllib/3.6" 80da4488-2424-11ea-837b-0242ac16000e 177245
influxdb_1 | [httpd] 172.22.0.3 - - [21/Dec/2019:19:02:58 +0000] "POST /write?db=metrics&precision=s HTTP/1.1" 204 0 "-" "Python-urllib/3.6" 80d75795-2424-11ea-837a-0242ac16000e 374255
sawtooth-validator-default-1 | [2019-12-21 19:03:01.169 ERROR threadpool] (Consensus) Unhandled exception during execution of task _HandlerManager.execute.<locals>.wrapped
sawtooth-validator-default-1 | Traceback (most recent call last):
sawtooth-validator-default-1 | File "/usr/lib/python3/dist-packages/sawtooth_validator/concurrent/threadpool.py", line 83, in wrapper
sawtooth-validator-default-1 | return_value = fn(*args, **kwargs)
sawtooth-validator-default-1 | File "/usr/lib/python3/dist-packages/sawtooth_validator/networking/dispatch.py", line 444, in wrapped
sawtooth-validator-default-1 | return callback(self._handler.handle(connection_id, message))
sawtooth-validator-default-1 | File "/usr/lib/python3/dist-packages/sawtooth_validator/consensus/handlers.py", line 378, in handle
sawtooth-validator-default-1 | self._proxy.validate_block(block_id)
sawtooth-validator-default-1 | File "/usr/lib/python3/dist-packages/sawtooth_validator/consensus/proxy.py", line 225, in validate_block
sawtooth-validator-default-1 | block = next(self._block_manager.get([block_id]))
sawtooth-validator-default-1 | File "/usr/lib/python3/dist-packages/sawtooth_validator/ffi.py", line 203, in __next__
sawtooth-validator-default-1 | payload = from_rust_vec(vec_ptr, vec_len, vec_cap)
sawtooth-validator-default-1 | File "/usr/lib/python3/dist-packages/sawtooth_validator/ffi.py", line 106, in from_rust_vec
sawtooth-validator-default-1 | py_bytes = bytes(vec_ptr[:vec_len.value])
sawtooth-validator-default-1 | MemoryError
influxdb_1 | [httpd] 172.22.0.2 - - [21/Dec/2019:19:03:01 +0000] "POST /write?db=metrics&precision=s HTTP/1.1" 204 0 "-" "Python-urllib/3.6" 82953819-2424-11ea-837c-0242ac16000e 23421
influxdb_1 | [httpd] 172.22.0.9 - - [21/Dec/2019:19:03:03 +0000] "POST /write?db=metrics&precision=s HTTP/1.1" 204 0 "-" "Python-urllib/3.6" 8383f49a-2424-11ea-837d-0242ac16000e 35783
sawtooth-settings-tp-default-0 | INFO | sawtooth_sdk::messag | Received Disconnect
sawtooth-settings-tp-default-0 | INFO | sawtooth_sdk::proces | Trying to Reconnect
sawtooth-settings-tp-default-0 | INFO | sawtooth_sdk::proces | connecting to endpoint: tcp://validator-0:4004
sawtooth-settings-tp-default-0 | INFO | sawtooth_sdk::proces | sending TpRegisterRequest: sawtooth_settings 1.0
sawtooth-poet-engine-0 | [2019-12-21 19:03:06.079 ERROR base_events] Task exception was never retrieved
sawtooth-poet-engine-0 | future: <Task finished coro=<_SendReceiveThread._monitor_disconnects() done, defined at /usr/lib/python3/dist-packages/sawtooth_sdk/messaging/stream.py:135> exception=CancelledError()>
sawtooth-poet-engine-0 | concurrent.futures._base.CancelledError
sawtooth-poet-engine-0 | [2019-12-21 19:03:06.127 ERROR base_events] Task exception was never retrieved
sawtooth-poet-engine-0 | future: <Task finished coro=<_SendReceiveThread._monitor_disconnects() done, defined at /usr/lib/python3/dist-packages/sawtooth_sdk/messaging/stream.py:135> exception=CancelledError()>
sawtooth-poet-engine-0 | concurrent.futures._base.CancelledError
sawtooth-poet-engine-0 | [2019-12-21 19:03:06.286 ERROR engine] Unhandled exception in message loop
sawtooth-poet-engine-0 | Traceback (most recent call last):
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_poet_engine/engine.py", line 192, in start
sawtooth-poet-engine-0 | self._try_to_publish()
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_poet_engine/engine.py", line 201, in _try_to_publish
sawtooth-poet-engine-0 | if self._initialize_block():
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_poet_engine/engine.py", line 63, in _initialize_block
sawtooth-poet-engine-0 | initialize = self._oracle.initialize_block(chain_head)
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_poet_engine/oracle.py", line 80, in initialize_block
sawtooth-poet-engine-0 | return self._publisher.initialize_block(block_header)
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_poet/poet_consensus/poet_block_publisher.py", line 448, in initialize_block
sawtooth-poet-engine-0 | poet_settings_view=poet_settings_view):
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_poet/poet_consensus/consensus_state.py", line 701, in validator_has_claimed_block_limit
sawtooth-poet-engine-0 | key_block_claim_limit = poet_settings_view.key_block_claim_limit
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_poet/poet_consensus/poet_settings_view.py", line 182, in key_block_claim_limit
sawtooth-poet-engine-0 | validate_function=lambda value: value > 0)
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_poet/poet_consensus/poet_settings_view.py", line 95, in _get_config_setting
sawtooth-poet-engine-0 | value_type=value_type)
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_poet/state/settings_view.py", line 75, in _get_setting
sawtooth-poet-engine-0 | SettingsView.setting_address(key))
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_poet_engine/oracle.py", line 267, in get
sawtooth-poet-engine-0 | addresses=[address])
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_sdk/consensus/zmq_service.py", line 333, in get_state
sawtooth-poet-engine-0 | response_type=response_type)
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_sdk/consensus/zmq_service.py", line 32, in _send
sawtooth-poet-engine-0 | ).result(self._timeout).content
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_sdk/messaging/future.py", line 38, in content
sawtooth-poet-engine-0 | raise ValidatorConnectionError()
sawtooth-poet-engine-0 | sawtooth_sdk.messaging.exceptions.ValidatorConnectionError: the connection to the validator was lost
sawtooth-poet-engine-0 | [2019-12-21 19:03:06.540 ERROR zmq_driver] Uncaught driver exception
sawtooth-poet-engine-0 | Traceback (most recent call last):
sawtooth-poet-engine-0 | File "/usr/lib/python3/dist-packages/sawtooth_sdk/consensus/zmq_driver.py", line 83, in _driver_loop
sawtooth-poet-engine-0 | message = future.result(1)
sawtooth-poet-engine-0 | File "/usr/lib/python3.6/concurrent/futures/_base.py", line 430, in result
sawtooth-poet-engine-0 | raise CancelledError()
sawtooth-poet-engine-0 | concurrent.futures._base.CancelledError
sawtooth-poet-validator-registry-tp-0 | Task exception was never retrieved
sawtooth-poet-validator-registry-tp-0 | future: <Task finished coro=<_SendReceiveThread._monitor_disconnects() done, defined at /usr/lib/python3/dist-packages/sawtooth_sdk/messaging/stream.py:135> exception=CancelledError()>
sawtooth-poet-validator-registry-tp-0 | concurrent.futures._base.CancelledError
sawtooth-validator-default-1 | [2019-12-21 19:03:07.304 INFO (unknown file)] [src/journal/chain.rs: 206] Building fork resolution for chain head 'Block(id: b4d266cb53f8760dbcc18fca0615b234ceb7dc944cd435722f7363af118bdf1455a01f41e103a16dc3983b0bbc2960f251fbf3102b977f216cf4758d79fdba48, block_num: 21, state_root_hash: 0b9c7c83b2f7534ba723c16f9e71d0a35f271f485a06c87480f5a4ec46963144, previous_block_id: e5e53884f13e15fe7474ec450150093b3f5976974c4a2529a25e8150d1519a921ae0606de70a63273680500954ace604665e8faaad701bad12caa84b8934aa8b)' against new block 'Block(id: 4a6a1342c829d81752127f5b95deaff1b9cac847ec3675714a2b1b086d281e6f44979beef8506041a89d841e089d8ae36e2513e3f65a91c2e634fa23a4052e06, block_num: 22, state_root_hash: 0b9c7c83b2f7534ba723c16f9e71d0a35f271f485a06c87480f5a4ec46963144, previous_block_id: b4d266cb53f8760dbcc18fca0615b234ceb7dc944cd435722f7363af118bdf1455a01f41e103a16dc3983b0bbc2960f251fbf3102b977f216cf4758d79fdba48)'
influxdb_1 | [httpd] 172.22.0.3 - - [21/Dec/2019:19:03:09 +0000] "POST /write?db=metrics&precision=s HTTP/1.1" 204 0 "-" "Python-urllib/3.6" 872e224f-2424-11ea-837e-0242ac16000e 68105
influxdb_1 | [httpd] 172.22.0.4 - - [21/Dec/2019:19:03:09 +0000] "POST /write?db=metrics&precision=s HTTP/1.1" 204 0 "-" "Python-urllib/3.6" 87491b1d-2424-11ea-837f-0242ac16000e 26546
sawtooth-validator-default-0 exited with code 137
sawtooth-rest-api-default-0 | [2019-12-21 19:03:09.607 WARNING route_handlers] Validator disconnected while waiting for response
influxdb_1 | [httpd] 172.22.0.5 - - [21/Dec/2019:19:03:09 +0000] "POST /write?db=metrics&precision=s HTTP/1.1" 204 0 "-" "Python-urllib/3.6" 876cfac6-2424-11ea-8380-0242ac16000e 95041
sawtooth-validator-default-2 | [2019-12-21 19:03:00.566 ERROR threadpool] (Consensus) Unhandled exception during execution of task _HandlerManager.execute.<locals>.wrapped
sawtooth-validator-default-2 | Traceback (most recent call last):
sawtooth-validator-default-2 | File "/usr/lib/python3/dist-packages/sawtooth_validator/concurrent/threadpool.py", line 83, in wrapper
sawtooth-validator-default-2 | return_value = fn(*args, **kwargs)
sawtooth-validator-default-2 | File "/usr/lib/python3/dist-packages/sawtooth_validator/networking/dispatch.py", line 444, in wrapped
sawtooth-validator-default-2 | return callback(self._handler.handle(connection_id, message))
sawtooth-validator-default-2 | File "/usr/lib/python3/dist-packages/sawtooth_validator/consensus/handlers.py", line 378, in handle
sawtooth-validator-default-2 | self._proxy.validate_block(block_id)
sawtooth-validator-default-2 | File "/usr/lib/python3/dist-packages/sawtooth_validator/consensus/proxy.py", line 225, in validate_block
sawtooth-validator-default-2 | block = next(self._block_manager.get([block_id]))
sawtooth-validator-default-2 | File "/usr/lib/python3/dist-packages/sawtooth_validator/ffi.py", line 203, in __next__
sawtooth-validator-default-2 | payload = from_rust_vec(vec_ptr, vec_len, vec_cap)
sawtooth-validator-default-2 | File "/usr/lib/python3/dist-packages/sawtooth_validator/ffi.py", line 106, in from_rust_vec
sawtooth-validator-default-2 | py_bytes = bytes(vec_ptr[:vec_len.value])
sawtooth-validator-default-2 | MemoryError
I am using Hyperledger Sawtooth 1.2 (chime), with the poet algorithm, with influxdb, and graphana. I created a script to evaluate the CPU and RAM consumption on the server, and observe the result in the image CPU and RAM consumption.
In addition, I took the system analyzed with the HTOP tool, and even the SWAP was fully consumed, as illustrated in this figure that shows the consumption moments before the error occurred.
In the Faq (https://sawtooth.hyperledger.org/faq/validator/) of doubts, I found the following statement:
Sawtooth stores the blockchain in a LMDB database at /var/lib/Sawtooth/block-00.lmdb . The LMDB database is a "sparse" file meaning no storage is allocated for the file until it is used (written to). The database should not run out of memory, as long as filesystem storage is available. The memory error could happen in Kubernetes or Docker or other virtual machine environments where there are no storage volumes mapped to the VM.
Since I am running my three validators in docker containers on the same server machine, I imagine the problem is that transactions are not being stored on disk but remain in RAM because of LMDB.
How to solve this problem?
