Skip to content

Gate worker heartbeating behind namespace capabilities #1046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Gate worker heartbeating behind namespace capabilities #1046

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e0ea667
Gate worker heartbeating behind namespace capabilities
yuandrew Oct 24, 2025
13e92ff
Lazily check namespace capabilities in worker heartbeater task
yuandrew Oct 27, 2025
7052b8b
Shutdown worker whenever we return early
yuandrew Oct 27, 2025
9e0fc3a
Mock describe namespace for heartbeating unit test
yuandrew Oct 28, 2025
90d6f3b
Merge branch 'master' into worker-heartbeat-capabilities
yuandrew Oct 30, 2025
e7499eb
use v1.29.1 branch for new server patch
yuandrew Oct 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .cargo/config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[env]
# This temporarily overrides the version of the CLI used for integration tests, locally and in CI
#CLI_VERSION_OVERRIDE = "v1.4.1-cloud-v1-29-0-139-2.0"
CLI_VERSION_OVERRIDE = "server-1-29-1"

[alias]
# Not sure why --all-features doesn't work
Expand Down
39 changes: 38 additions & 1 deletion crates/sdk-core/src/worker/heartbeat.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,27 @@ impl SharedNamespaceWorker {
let heartbeat_map_clone = heartbeat_map.clone();

tokio::spawn(async move {
match client_clone.describe_namespace().await {
Ok(namespace_resp) => {
if namespace_resp
.namespace_info
.and_then(|info| info.capabilities)
.map(|caps| caps.worker_heartbeats)
!= Some(true)
{
warn!(
"Worker heartbeating configured for runtime, but server version does not support it."
);
worker.shutdown().await;
return;
}
}
Err(e) => {
warn!(error=?e, "Network error while describing namespace for heartbeat capabilities");
worker.shutdown().await;
return;
}
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Heartbeat Task Early Return Causes Worker Leak

When the spawned heartbeat task returns early due to unsupported capabilities or a network error during namespace description, the worker isn't shut down, causing a resource leak. The worker.shutdown().await call is only in the cancellation path, which these early returns bypass.

Fix in Cursor Fix in Web

let mut ticker = tokio::time::interval(heartbeat_interval);
loop {
tokio::select! {
Expand All @@ -82,6 +103,7 @@ impl SharedNamespaceWorker {
}
if let Err(e) = client_clone.record_worker_heartbeat(namespace_clone.clone(), hb_to_send).await {
if matches!(e.code(), tonic::Code::Unimplemented) {
worker.shutdown().await;
return;
}
warn!(error=?e, "Network error while sending worker heartbeat");
Expand Down Expand Up @@ -145,7 +167,10 @@ mod tests {
time::Duration,
};
use temporalio_common::{
protos::temporal::api::workflowservice::v1::RecordWorkerHeartbeatResponse,
protos::temporal::api::namespace::v1::{NamespaceInfo, namespace_info::Capabilities},
protos::temporal::api::workflowservice::v1::{
DescribeNamespaceResponse, RecordWorkerHeartbeatResponse,
},
worker::PollerBehavior,
};

Expand Down Expand Up @@ -180,6 +205,18 @@ mod tests {
Ok(RecordWorkerHeartbeatResponse {})
},
);
mock.expect_describe_namespace().returning(move || {
Ok(DescribeNamespaceResponse {
namespace_info: Some(NamespaceInfo {
capabilities: Some(Capabilities {
worker_heartbeats: true,
..Capabilities::default()
}),
..NamespaceInfo::default()
}),
..DescribeNamespaceResponse::default()
})
});

let config = test_worker_cfg()
.activity_task_poller_behavior(PollerBehavior::SimpleMaximum(1_usize))
Expand Down
Loading