1

I want to understand how cadence-client manages memory for long running workflows. Consider that a workflow runs for 6 months, but it is not active for this entire duration. It becomes active when the client receives a signal related to this workflow, performs some activities and then again become idle. I am using Workflow.await method given by java library to acheive this.

My question is how does cadence manage these idle workflows(and all the state variables that this workflow creates)? Since the workflow has not completed yet, will it allow java garbage collector to remove the classes related to the workflow or will they remain in the heap space consuming memory and CPU? Can we remove the idle workflows from memory so that our memory footprint does not increase ?

Ezio
  • 2,837
  • 2
  • 29
  • 45

1 Answers1

1

Assuming the memory you are talking here is about Cadence workflow workers.

how does cadence manage these idle workflows(and all the state variables that this workflow creates)? Since the workflow has not completed yet, will it allow java garbage collector to remove the classes related to the workflow or will they remain in the heap space consuming memory and CPU

By default, the idle workflows will be kept in memory as "sticky cache" until the cache is full and evicted by LRU. This is to save the performance & IO of workflow decision task(or workflow task in Temporal) having to replay the history to rebuild the memory states. Garbage collector won't be able to release them unless the workflow is evicted from cache.

Can we remove the idle workflows from memory so that our memory footprint does not increase ?

Yes, you can disable the stickyCache in WorkerOptions(or WorkerFactoryOptions) so that there is no memory overhead for this.

We should improve this in the SDK -- The cacheSize is by number of workflow threads and there is no config as total size of memory consumed. This makes it very hard to manage the memory. The only way today is to monitor on avg historySize.

Also, there is no proactive pruning mechanism to evict workflows today. Ideally we should let users configure a total cache size, and evict the workflows when the memory consumption is exceeding the configured size. I open an issue to track this: https://github.com/uber/cadence-java-client/issues/671

Long Quanzheng
  • 2,076
  • 1
  • 10
  • 22