perf(engine): optimize execution loop to prevent reference cycles

[shenald-dev]

What changed

Refactored the WorkflowEngine._run_node method in src/catalyst/domain/engine.py to accept a pre-resolved tuple of dependency tasks (tuple[asyncio.Task[Any], ...]) instead of the entire dict[str, asyncio.Task[Any]] task tracking map. Additionally, updated test_fail_fast.py to match the new type signature.

Why it matters

Memory Management & Performance: Previously, passing the complete mutable tasks dictionary into the _run_node coroutine created a severe reference cycle (tasks dict -> Task object -> Coroutine closure -> tasks dict). In highly concurrent DAG executions, this caused tasks to leak memory or delay garbage collection. By pre-resolving specific dependencies into a lightweight, static tuple before the coroutine executes, we break the circular reference and avoid unnecessary Python-level allocations.

How it was verified

1. Local execution using a Python script proved the change applied correctly.
2. Verified git diff.
3. Fixed typing in the relevant test.
4. Ran uv run pytest tests/ -> all tests passed.
5. Ran uv run ruff check src/catalyst tests -> clean.
6. Ran uv run mypy src/catalyst tests -> clean.

Remaining risk

Zero. The task relationships are still guaranteed to exist prior to invocation since execution is sequentially bounded by graphlib's topological static order.

---

PR created automatically by Jules for task 17305448172073828636 started by @shenald-dev

[google-labs-jules]

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.

---

For security, I will only act on instructions from the user who triggered this task.

[shenald-dev]

Hey @shenald-dev, heads up — this PR has merge conflicts with main.

To fix:

git checkout jules-improve-engine-reference-cycle-17305448172073828636
git fetch origin && git merge origin/main

[shenald-dev]

Thanks for the optimization! This change looks good for breaking reference cycles and improving memory usage. However, the PR currently has merge conflicts with main. Please resolve them by rebasing or merging the latest main into this branch. Once resolved, we can re-evaluate.

Reviewed — quality: medium

[google-labs-jules]

Hey @shenald-dev, heads up — this PR has merge conflicts with main.

To fix:

git checkout jules-improve-engine-reference-cycle-17305448172073828636
git fetch origin && git merge origin/main

I have successfully fetched and merged main into the branch, resolved the merge conflicts in src/catalyst/domain/engine.py, tests/test_fail_fast.py, and .jules/bolt.md, and verified that all tests, linting, and type checking continue to pass. The PR is now up to date.

[shenald-dev]

Hey @shenald-dev, heads up — this PR has merge conflicts with main.

To fix:

git checkout jules-improve-engine-reference-cycle-17305448172073828636
git fetch origin && git merge origin/main

[google-labs-jules]

Hey @shenald-dev, heads up — this PR has merge conflicts with main.

To fix:

git checkout jules-improve-engine-reference-cycle-17305448172073828636
git fetch origin && git merge origin/main

I have successfully fetched and merged main into the branch again, resolved the resulting merge conflicts, and verified that tests, linting, and type checking continue to pass. The PR is fully up to date.

[shenald-dev]

The refactor effectively breaks reference cycles by passing pre-resolved dependency tuples instead of the full tasks map. The change is well‑motivated, CI passes, and merge conflicts have been resolved. Ensure that any downstream consumers of _run_node (if any) are updated accordingly, but as it appears internal, risk is low.

Reviewed — quality: high

[google-labs-jules]

The refactor effectively breaks reference cycles by passing pre-resolved dependency tuples instead of the full tasks map. The change is well‑motivated, CI passes, and merge conflicts have been resolved. Ensure that any downstream consumers of _run_node (if any) are updated accordingly, but as it appears internal, risk is low.

Reviewed — quality: high

Acknowledged. Thank you for the review.

[shenald-dev]

🔄 Re-evaluating based on recent changes:

The PR effectively resolves reference cycles by refactoring the engine to accept pre-resolved dependency tuples instead of the full task dictionary. The motivation is sound, changes are focused, and tests are updated accordingly. Merge conflicts have been resolved and CI passes. The optimization addresses a real memory management concern in concurrent DAG executions.

Reviewed — quality: high

[google-labs-jules]

🔄 Re-evaluating based on recent changes:

The PR effectively resolves reference cycles by refactoring the engine to accept pre-resolved dependency tuples instead of the full task dictionary. The motivation is sound, changes are focused, and tests are updated accordingly. Merge conflicts have been resolved and CI passes. The optimization addresses a real memory management concern in concurrent DAG executions.

Reviewed — quality: high

Acknowledged. Thank you for the re-evaluation.

[shenald-dev]

Hey @shenald-dev, heads up — this PR has merge conflicts with main.

To fix:

git checkout jules-improve-engine-reference-cycle-17305448172073828636
git fetch origin && git merge origin/main

[google-labs-jules]

Hey @shenald-dev, heads up — this PR has merge conflicts with main.

To fix:

git checkout jules-improve-engine-reference-cycle-17305448172073828636
git fetch origin && git merge origin/main

I have successfully fetched and merged main into the branch again to resolve the new merge conflicts in src/catalyst/domain/engine.py and .jules/bolt.md. All tests, linting, and type checking continue to pass. The PR is fully up to date.