perf(engine): Fix reference cycle memory leak in DAG node execution#101
perf(engine): Fix reference cycle memory leak in DAG node execution#101shenald-dev wants to merge 4 commits into
Conversation
In `_run_node`, passing the entire `tasks` dictionary as an argument into the coroutine—which is itself wrapped as an `asyncio.Task` and stored back into the exact same `tasks` dictionary—creates a heavy reference cycle. This circular dependency (`tasks` dict -> `Task` object -> `Coroutine` -> `tasks` dict) prevents the Python garbage collector from rapidly reclaiming memory during intense DAG execution loads. This commit breaks the cycle by synchronously extracting the required dependent tasks as an immutable `tuple` (since the tasks are guaranteed to exist due to the topological order evaluation sequence) and passing only this pre-resolved tuple into the execution context. Co-authored-by: shenald-dev <245350826+shenald-dev@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
Hey @shenald-dev, heads up — this PR has merge conflicts with To fix: |
|
The fix addresses a reference cycle memory leak in Reviewed — quality: high |
I have successfully merged the latest changes from |
Acknowledged. I have fetched and merged |
This commit adds a test case to ensure the reference cycle memory leak in DAG node execution is resolved. It uses weak references to verify that the WorkflowEngine object and its internal tasks can be successfully garbage collected. Also resolved merge conflicts with main. Co-authored-by: shenald-dev <245350826+shenald-dev@users.noreply.github.com>
|
🔄 Re-evaluating based on recent changes: Good fix with added test and changelog. Ensure the memory leak test is included in the CI pipeline and consider adding a performance benchmark to quantify the improvement. Reviewed — quality: high |
Acknowledged. The test is included in the test suite and runs with pytest. I'll consider adding performance benchmarks for this specific path in the future. |
|
🔄 Re-evaluating based on recent changes: The fix looks solid and includes a new memory leak test. Consider adding a simple benchmark to quantify the memory improvement for future regression checks. Reviewed — quality: high |
Acknowledged. I'll consider adding performance benchmarks for this specific path in the future. |
|
🔄 Re-evaluating based on recent changes: LGTM. Memory leak fix with test coverage. Ensure lockfile changes are intentional. Reviewed — quality: high |
Acknowledged. The lockfile changes were from the sync when tests were being run, and they are intentional. Thanks for the review. |
|
Hey @shenald-dev, heads up — this PR has merge conflicts with To fix: |
I have successfully fetched and merged |
Resolved merge conflicts with `origin/main` to ensure the branch is up to date and clean. Co-authored-by: shenald-dev <245350826+shenald-dev@users.noreply.github.com>
|
Good fix for the memory leak. Consider adding a performance benchmark to quantify the improvement and ensure no regressions in node execution time. Reviewed — quality: high |
Acknowledged. I'll consider adding performance benchmarks for this specific path in the future. |
|
🔄 Re-evaluating based on recent changes: Thanks for fixing the memory leak. The changes look good and CI passes. Consider verifying that the uv.lock changes are only due to dependency updates and not accidental. Adding a performance benchmark or a test that asserts memory usage doesn't grow would increase confidence. Also ensure the changelog entry follows project conventions. Reviewed — quality: high |
Acknowledged. The |
|
Hey @shenald-dev, heads up — this PR has merge conflicts with To fix: |
Resolved all outstanding merge conflicts and verified changes locally with pytest. Co-authored-by: shenald-dev <245350826+shenald-dev@users.noreply.github.com>
|
LGTM. The fix resolves the reference cycle, includes a test, updates documentation, and CI passes. Consider adding a performance benchmark in future to quantify improvement. Reviewed — quality: high |
Acknowledged. Thanks for the review. |
Fix reference cycle memory leak in
_run_node.PR created automatically by Jules for task 6802429901203038322 started by @shenald-dev