The specific role of worker in DolphinDB?

Question

What does worker, secondaryWorker, web worker, infra worker, dynamic worker, and local executors stand for respectively in DolphinDB?  Why the secondaryWorker and dynamic worker is introduced and what’s the usage?

Answer 1

worker: the thread of regular interactive jobs. It divides clients’ requests into subtasks once received. Depending on the task granularity, the worker will either execute the tasks or allocate them to local executor or remote executor. The number of workers can be set by specifying the configuration parameter workerNum and the default value is determined by the number of CPU cores.

secondary worker: the thread of secondary jobs. It is used to avoid job loops and solve the deadlocks caused by circular dependency between tasks. The upper limit can be set by specifying the configuration parameter secondaryWorkerNum and the default value is workerNum.

web worker: the thread that processes HTTP requests. DolphinDB provides a web interface for cluster management, allowing users to interact with DolphinDB nodes. The upper limit can be set by specifying the configuration parameter webWorkerNum and the default value is 1.

infra worker: the thread that reports heartbeat within clusters. It solves the problem that the heartbeat cannot be reported to the master in time when a cluster is under high pressure.

dynamic worker: the dynamic working thread as a supplemental to worker. If a new task is requested when all the worker threads are occupied, the system creates a dynamic worker thread to perform the task. The upper limit can be set by specifying the configuration parameter maxDynamicWorker and the default value is workerNum. The thread will be recycled by the system after being idle for 60 seconds to release memory resources.

local executor: the local thread that executes sub-tasks allocated by worker. Each local executor can only execute one task at a time. All worker threads share one local executor. The number of local executors can be set by specifying configuration parameter localExecutors and the default value is the number of CPU cores minus 1. The number of workers and local executors directly determines the system’s performance for concurrent computing.