test: cover push scheduling in client integration tests#1685
test: cover push scheduling in client integration tests#1685abhinavgautam01 wants to merge 4 commits into
Conversation
- Add standalone scheduler/executor APIs for TaskSchedulingPolicy. - Run push-staged executor path in tests; fix scheduler endpoint and gRPC readiness. - Extend rstest fixtures and context_setup with push cases.
1f1ec15 to
3a70d60
Compare
|
You added standalone push scheduler, but this is not covered by tests ATM ? |
added coverage in |
Sorry, not quite what I meant. I meant having 0 change on the test cases themselves, and just splitting what you did to #[fixture]
async fn remote(
#[values(TaskSchedulingPolicy::PullStaged, TaskSchedulingPolicy::PushStaged)]
policy: TaskSchedulingPolicy,
) -> SessionContext {
common::remote_context_with_scheduling(policy).await
}and the same for Also now you have diverging code style with |
|
updated... i removed the standalone one-off test and changed the context tests to keep the two context cases ( i also aligned |
Which issue does this PR close?
Closes #1463.
Rationale for this change
Issue #1463 asked for integration tests to cover push-staged scheduling as well as pull-staged. The in-process test cluster was effectively pull-only: the standalone scheduler used a fixed identity while listening on a random port, and the standalone executor always used the pull
poll_loop. That gap meant we did not exercise the same scheduler/executor path as the default push policy in real deployments.In addition, exposing push mode in tests surfaced a real correctness issue: push executors report task status via gRPC clients keyed by the scheduler’s advertised
host:port; a hard-coded placeholder did not match the actual listener, which could cause hangs or failed status updates.What changes are included in this PR?
standalone): bind127.0.0.1:0first, then build the in-memory cluster /SchedulerServerwithscheduler_endpoint = addr.to_string()so task metadata and curator IDs match the real gRPC listener.standalone): add push-staged support using the existingexecutor_server::startuppath; keep pull-staged behavior onpoll_loopas before.executor_server::startup: wait until the executor gRPC address accepts a TCP connection beforeregister_executor, reducing a race with asynchronously started listeners.rstestcases for push (remote_push,remote_state_push);context_setupremote tests parameterized overTaskSchedulingPolicy; align test URLs with127.0.0.1for consistency with the binder.127.0.0.1host alignment for cluster URLs.grpc_porttou32for protobuf registration fields.Tests:
cargo test -p ballista --features standalone(withprotocavailable).Are there any user-facing changes?
Yes, but small and additive. New public APIs were added on the scheduler and executor
standalonemodules (policy-aware constructors /_with_scheduling_policyhelpers). Existing entry points retain prior behavior where applicable (e.g. default standalone scheduler helpers still target pull-staged integration defaults as before).No changes to CLI flags or documented user workflows beyond optional use of these APIs by embedders/tests.