[Bug] temporal_worker_task_slots_used (sum across worker types) never reaches 0 during inter-activity gaps

[Sanil2108]

What are you really trying to do?

We're using KEDA to autoscale Temporal Python SDK workers to zero replicas when no workflows are executing. We use temporal_worker_task_slots_used (summed across all worker types) as the KEDA composite metric trigger, expecting it to be 0 when the worker is idle.

Describe the bug

sum(temporal_worker_task_slots_used{task_queue="..."}) never drops to 0 even when no workflows are running

Minimal Reproduction

Start a Python SDK worker with any workflow that has ≥2 sequential activities
Scrape temporal_worker_task_slots_used at 15s intervals
Observe: WorkflowWorker slots = 1 during WFT scheduling between activity 1 completing and activity 2 starting — even though ActivityWorker = 0 at that exact moment
Sum across all worker_type values: the sum is always ≥1 while any workflow is in progress, including the inter-activity scheduling gaps

Environment/Versions

OS: Linux (Kubernetes, Python 3.13)
Temporal Server: self-hosted, version does not have worker_heartbeats capability (pre-1.27)
Python SDK: latest (temporalio with sdk-core at commit 2872b53)
Worker Deployment versioning: enabled (WorkerVersioningStrategy::WorkerDeploymentBased)
Deployment: Kubernetes with KEDA autoscaler, cooldownPeriod: 600s, activationTargetValue: 0

Additional context

[tconley1428]

Can you clarify a little bit?

WorkflowWorker slots = 1 during WFT scheduling between activity 1 completing and activity 2 starting — even though ActivityWorker = 0 at that exact moment

Yes, this is entirely possible. Workflow slots are occupied by the workflow worker executing a workflow task, which is exactly what happens between activity 1 completing and activity 2 being scheduled.

WorkflowWorker slots = 1 during WFT scheduling between activity 1 completing and activity 2 starting — even though ActivityWorker = 0 at that exact moment

There's nothing unexpected in that statement. Why do you think 0 activity slots occupied implies that there should be 0 workflow slots occupied?

From the sounds of this workflow, it is entirely expected that the slots would never drop to 0.

1. The first workflow task runs, schedules the first activity - 1 workflow slot used
2. The first activity is running - 1 activity slot used
3. The first activity completes, another workflow task runs, schedules the second activity - 1 workflow slot used
4. The second activity is running - 1 activity slot used
5. The second activity completes, another workflow task runs - 1 workflow slot used
6. Workflow completes - No slots used

There are potentially zero slots used at some of the boundaries between these states depending on the order the worker receives things, but not seeing those on a 15s interval poll isn't surprising.

[Sanil2108]

Can you clarify a little bit?

Sure, to clarify more, we have no workflows running, and the worker is completely idle, however the worker is still emitting the metric temporal_worker_task_slots_used which is oscillating between 0 and 1. We observed this on multiple temporal clusters, even hours after the last workflow had completed.