I would like to run Thingsboard in the microservice configuration on a AWS EC2 machine but when I try to launch the services the CPU goes at 100% and the machine crashes. At first I thought that the t3.medium was too small to run all the services (I'm using the default configuration in the git file) so I went with a m6a.large.
Now it can go pass the start of the services but freezes after a couple of minutes. I can even access the localhost but all i get is a 503 before everything stop working.
I've started to think there is some king of looping error blocking everything, this is a fresh machine with no data on it, so it seams strage to me this much need for resources.
I've run one time with the logs open and I saw there are some error in the kafkajs / js-executor phase, can you help me understand if this is the problem and how to fix it?
The following errors are recurring and present every instance of a js-executor. the last minute or so of logs where this errors in loop with other infos and warning.
docker-tb-js-executor-5 | 2023-08-21 09:22:05,716 [kafkajs] ERROR: Connection error: connect ECONNREFUSED IP:9092: {"timestamp":"2023-08-21T09:22:05.716Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","stack":"Error: connect ECONNREFUSED IP:9092\n at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)"}
docker-tb-js-executor-5 | 2023-08-21 09:22:05,716 [kafkajs] ERROR: Failed to connect to seed broker, trying another broker from the list: Connection error: connect ECONNREFUSED IP:9092: {"timestamp":"2023-08-21T09:22:05.716Z","logger":"kafkajs","retryCount":4,"retryTime":3380}
docker-tb-js-executor-5 | 2023-08-21 09:22:10,099 [kafkajs] ERROR: Connection timeout: {"timestamp":"2023-08-21T09:22:10.098Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs"}
docker-tb-js-executor-5 | 2023-08-21 09:22:10,099 [kafkajs] ERROR: Failed to connect to seed broker, trying another broker from the list: Connection timeout: {"timestamp":"2023-08-21T09:22:10.099Z","logger":"kafkajs","retryCount":5,"retryTime":5622}
docker-tb-js-executor-5 | 2023-08-21 09:22:10,100 [main] ERROR: Failed to start ThingsBoard JavaScript Executor Microservice: Connection timeout
docker-tb-js-executor-5 | 2023-08-21 09:22:10,100 [main] ERROR: KafkaJSNonRetriableError
docker-tb-js-executor-5 | Caused by: KafkaJSConnectionError: Connection timeout
docker-tb-js-executor-5 | at Timeout.onTimeout [as _onTimeout] (/usr/share/tb-js-executor/node_modules/kafkajs/src/network/connection.js:223:23)
docker-tb-js-executor-5 | at listOnTimeout (node:internal/timers:559:17)
docker-tb-js-executor-5 | at processTimers (node:internal/timers:502:7)
docker-tb-js-executor-5 | 2023-08-21 09:22:10,100 [main] INFO: Exiting with status: -1 ...
followed by
docker-tb-js-executor-5 | 2023-08-21 09:22:17,269 [main] INFO: ===CONFIG END===
docker-tb-js-executor-5 | 2023-08-21 09:22:17,269 [main] INFO: Starting ThingsBoard JavaScript Executor Microservice...
docker-tb-js-executor-5 | 2023-08-21 09:22:17,271 [main] INFO: Starting Kafka template...
docker-tb-js-executor-5 | 2023-08-21 09:22:17,273 [kafkaTemplate] INFO: Kafka Bootstrap Servers: kafka:9092
docker-tb-js-executor-5 | 2023-08-21 09:22:17,275 [kafkaTemplate] INFO: Kafka Requests Topic: js_eval.requests
docker-tb-js-executor-5 | 2023-08-21 09:22:17,458 [kafkajs] ERROR: Response CreateTopics(key: 19, version: 3): {"timestamp":"2023-08-21T09:22:17.458Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","error":"Topic creation errors","correlationId":4,"size":74}
docker-tb-js-executor-5 | 2023-08-21 09:22:17,556 [kafkajs] ERROR: Response Metadata(key: 3, version: 6): {"timestamp":"2023-08-21T09:22:17.556Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","error":"This server does not host this topic-partition","correlationId":1,"size":86}
docker-tb-js-executor-5 | 2023-08-21 09:22:17,557 [main] ERROR: Failed to start ThingsBoard JavaScript Executor Microservice: This server does not host this topic-partition
docker-tb-js-executor-5 | 2023-08-21 09:22:17,557 [main] ERROR: KafkaJSProtocolError: This server does not host this topic-partition
docker-tb-js-executor-5 | at createErrorFromCode (/usr/share/tb-js-executor/node_modules/kafkajs/src/protocol/error.js:581:10)
docker-tb-js-executor-5 | at Object.parse (/usr/share/tb-js-executor/node_modules/kafkajs/src/protocol/requests/metadata/v0/response.js:55:11)
docker-tb-js-executor-5 | at Connection.send (/usr/share/tb-js-executor/node_modules/kafkajs/src/network/connection.js:433:35)
docker-tb-js-executor-5 | at processTicksAndRejections (node:internal/process/task_queues:96:5)
docker-tb-js-executor-5 | at async Broker.[private:Broker:sendRequest] (/usr/share/tb-js-executor/node_modules/kafkajs/src/broker/index.js:875:14)
docker-tb-js-executor-5 | at async Broker.metadata (/usr/share/tb-js-executor/node_modules/kafkajs/src/broker/index.js:177:12)
docker-tb-js-executor-5 | at async /usr/share/tb-js-executor/node_modules/kafkajs/src/cluster/brokerPool.js:158:25
docker-tb-js-executor-5 | at async /usr/share/tb-js-executor/node_modules/kafkajs/src/cluster/index.js:111:14
docker-tb-js-executor-5 | at async Cluster.refreshMetadata (/usr/share/tb-js-executor/node_modules/kafkajs/src/cluster/index.js:172:5)
docker-tb-js-executor-5 | at async Cluster.addMultipleTargetTopics (/usr/share/tb-js-executor/node_modules/kafkajs/src/cluster/index.js:230:11)
docker-tb-js-executor-5 | 2023-08-21 09:22:17,558 [main] INFO: Exiting with status: -1 ..
some other errors further down the logs
docker-tb-js-executor-5 | 2023-08-21 09:31:07,811 [kafkajs] ERROR: Response Metadata(key: 3, version: 6): {"timestamp":"2023-08-21T09:31:07.810Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","error":"There is no leader for this topic-partition as we are in the middle of a leadership election","correlationId":1,"size":6431}
docker-tb-js-executor-5 | 2023-08-21 09:31:08,134 [kafkajs] ERROR: Response Metadata(key: 3, version: 6): {"timestamp":"2023-08-21T09:31:08.134Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","error":"There is no leader for this topic-partition as we are in the middle of a leadership election","correlationId":2,"size":6431}
docker-tb-js-executor-5 | 2023-08-21 09:31:08,869 [kafkajs] ERROR: Response Metadata(key: 3, version: 6): {"timestamp":"2023-08-21T09:31:08.869Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","error":"There is no leader for this topic-partition as we are in the middle of a leadership election","correlationId":3,"size":6431}
docker-tb-js-executor-5 | 2023-08-21 09:31:10,069 [kafkajs] ERROR: Response Metadata(key: 3, version: 6): {"timestamp":"2023-08-21T09:31:10.069Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","error":"There is no leader for this topic-partition as we are in the middle of a leadership election","correlationId":4,"size":6431}
docker-tb-js-executor-5 | 2023-08-21 09:31:12,217 [kafkajs] ERROR: Response Metadata(key: 3, version: 6): {"timestamp":"2023-08-21T09:31:12.217Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","error":"There is no leader for this topic-partition as we are in the middle of a leadership election","correlationId":5,"size":6431}
and
docker-tb-js-executor-5 | 2023-08-21 09:32:21,468 [kafkajs] ERROR: Connection error: connect ECONNREFUSED IP:9092: {"timestamp":"2023-08-21T09:32:21.467Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","stack":"Error: connect ECONNREFUSED IP:9092\n at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)"}
docker-tb-js-executor-5 | 2023-08-21 09:32:21,469 [kafkajs] ERROR: Failed to connect to seed broker, trying another broker from the list: Connection error: connect ECONNREFUSED IP:9092: {"timestamp":"2023-08-21T09:32:21.469Z","logger":"kafkajs","retryCount":0,"retryTime":339}
docker-tb-js-executor-5 | 2023-08-21 09:32:21,811 [kafkajs] ERROR: Connection error: connect ECONNREFUSED IP:9092: {"timestamp":"2023-08-21T09:32:21.811Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","stack":"Error: connect ECONNREFUSED IP:9092\n at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)"}
docker-tb-js-executor-5 | 2023-08-21 09:32:21,812 [kafkajs] ERROR: Failed to connect to seed broker, trying another broker from the list: Connection error: connect ECONNREFUSED IP:9092: {"timestamp":"2023-08-21T09:32:21.812Z","logger":"kafkajs","retryCount":1,"retryTime":710}
docker-tb-js-executor-5 | 2023-08-21 09:32:22,524 [kafkajs] ERROR: Connection error: connect ECONNREFUSED IP:9092: {"timestamp":"2023-08-21T09:32:22.524Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","stack":"Error: connect ECONNREFUSED IP:9092\n at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)"}
docker-tb-js-executor-5 | 2023-08-21 09:32:22,525 [kafkajs] ERROR: Failed to connect to seed broker, trying another broker from the list: Connection error: connect ECONNREFUSED IP:9092: {"timestamp":"2023-08-21T09:32:22.525Z","logger":"kafkajs","retryCount":2,"retryTime":1362}
docker-tb-js-executor-5 | 2023-08-21 09:32:23,889 [kafkajs] ERROR: Connection error: connect ECONNREFUSED IP:9092: {"timestamp":"2023-08-21T09:32:23.889Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","stack":"Error: connect ECONNREFUSED IP:9092\n at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)"}
docker-tb-js-executor-5 | 2023-08-21 09:32:23,890 [kafkajs] ERROR: Failed to connect to seed broker, trying another broker from the list: Connection error: connect ECONNREFUSED IP:9092: {"timestamp":"2023-08-21T09:32:23.890Z","logger":"kafkajs","retryCount":3,"retryTime":2916}
docker-tb-js-executor-5 | 2023-08-21 09:32:26,820 [kafkajs] ERROR: Connection error: connect ECONNREFUSEDIP:9092: {"timestamp":"2023-08-21T09:32:26.820Z","logger":"kafkajs","broker":"kafka:9092","clientId":"kafkajs","stack":"Error: connect ECONNREFUSED IP:9092\n at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1247:16)"}
docker-tb-js-executor-5 | 2023-08-21 09:32:26,821 [kafkajs] ERROR: Failed to connect to seed broker, trying another broker from the list: Connection error: connect ECONNREFUSED IP:9092: {"timestamp":"2023-08-21T09:32:26.821Z","logger":"kafkajs","retryCount":4,"retryTime":6118}
I don't seem to have any errors with the others services.
Thank you for your time!
