Python: information-flow control prompt injection defense

[eavanvalkenburg]

Motivation and Context

This draft PR brings the Python information-flow control prompt injection defense work from feature/python-fides toward main for review and integration.

Description

• adds the FIDES information-flow control security model for tracking integrity and confidentiality through agent execution
• introduces secure-agent middleware, secure tools, and context-provider-based configuration for policy enforcement and hidden-content handling
• includes supporting samples, docs, and follow-up fixes needed to keep the Python workspace validation suite green

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the Contribution Guidelines
• All unit tests pass, and I have added new tests where possible
• Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

[moonbox3]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/core/agent_framework
_feature_stage.py	116	12	89%	98, 155, 162, 164–166, 173, 175, 180, 187, 206, 234
_tools.py	991	83	91%	192–193, 366, 368, 381, 406–408, 416, 434, 448, 455, 462, 485, 487, 494, 502, 585, 589, 621–623, 631, 676–678, 680, 703, 729, 733, 771–773, 777, 799, 941–947, 983, 995, 997, 999, 1002–1005, 1026, 1030, 1034, 1048–1050, 1397, 1421, 1483, 1511, 1533, 1537, 1667, 1671, 1717, 1778–1779, 1882, 1935, 1955, 1957, 2013, 2076, 2248–2249, 2269, 2325–2326, 2464–2465, 2532, 2537, 2544
observability.py	755	84	88%	377, 379–380, 383, 386, 389–390, 395–396, 402–403, 409–410, 417, 419–421, 424–426, 431–432, 438–439, 445–446, 453, 610–611, 739, 743–745, 747, 751–752, 756, 794, 796, 807–809, 811–813, 817, 825, 949–950, 1112, 1355–1356, 1454–1459, 1466–1469, 1473–1481, 1488, 1553–1557, 1616–1617, 1751, 1843, 2040, 2258, 2260
security.py	722	173	76%	437–438, 456, 459–460, 538–539, 541, 589, 601, 617, 634, 701–702, 705, 709, 716–717, 719, 726–731, 733, 735–742, 744, 747–751, 753–755, 757–758, 761–762, 764–765, 768–770, 772–776, 779–783, 785, 1037, 1043–1044, 1049, 1051–1052, 1082–1083, 1111–1119, 1140, 1243–1244, 1275–1276, 1301–1302, 1306–1307, 1341–1344, 1426–1427, 1432–1437, 1442–1444, 1775, 1780–1781, 1785, 1789–1791, 1840, 1852, 1854, 1871, 1892, 1903, 1929, 1970–1971, 1996, 2093, 2121–2123, 2158–2160, 2168, 2176, 2423–2424, 2426, 2428–2430, 2433, 2445, 2451–2452, 2454, 2485, 2489–2492, 2494, 2608, 2611–2614, 2617–2618, 2620, 2622, 2624, 2627–2631, 2638, 2642, 2653–2654, 2656, 2658–2660, 2721, 2731–2732
TOTAL	29771	3643	87%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
5883	30 💤	0 ❌	0 🔥	1m 37s ⏱️

[Copilot]

Pull request overview

This PR brings the Python FIDES information-flow control (integrity + confidentiality) prompt-injection/data-exfiltration defense closer to main, adding core security primitives/middleware, DevUI support for policy-violation approvals, and end-to-end samples/docs/tests.

Changes:

• Adds FIDES security samples and documentation (quick start + developer guide + implementation summary + ADR).
• Extends DevUI mapping/execution to round-trip policy-violation metadata through approval events.
• Adjusts core function-invocation plumbing and test suites to support the new approval/policy flow and keep validation green.

Reviewed changes

Copilot reviewed 14 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
python/samples/02-agents/security/repo_confidentiality_example.py	New sample demonstrating confidentiality labels + exfiltration prevention.
python/samples/02-agents/security/email_security_example.py	New sample demonstrating integrity taint handling + quarantined processing.
python/samples/02-agents/security/README.md	New quick-start documentation for SecureAgentConfig/FIDES patterns.
python/samples/02-agents/security/FIDES_DEVELOPER_GUIDE.md	New deep-dive developer guide for the FIDES model and APIs.
python/packages/core/tests/test_security.py	New comprehensive unit tests for labeling/middleware/policy enforcement.
python/packages/core/agent_framework/_tools.py	Updates tool invocation + approval replacement behavior for policy approvals.
python/packages/core/agent_framework/init.py	Exports new FIDES/security APIs and ai_function alias.
python/packages/core/agent_framework/_feature_stage.py	Adds ExperimentalFeature.FIDES.
python/packages/core/agent_framework/observability.py	Hardens finish_reason attribute handling.
python/packages/devui/agent_framework_devui/_mapper.py	Emits policy violation details in approval request events when present.
python/packages/devui/agent_framework_devui/_executor.py	Preserves policy-violation metadata on approval responses.
python/packages/devui/tests/devui/test_ui_memory_regression.py	Skips the test when CDP websocket URL cannot be obtained.
python/packages/foundry/tests/foundry/test_foundry_embedding_client.py	Makes env patching deterministic via clear=True.
docs/features/FIDES_IMPLEMENTATION_SUMMARY.md	Adds an implementation summary for FIDES components/deliverables.
docs/decisions/0024-prompt-injection-defense.md	New ADR documenting the design rationale and tradeoffs.

[github-actions]

Automated Code Review

Reviewers: 1 | Confidence: 88%

✗ Design Approach

The approval-flow work mostly moves in the right direction—matching approved results by call_id is a solid improvement—but one new piece of the design is too brittle: _replace_approval_contents_with_results now detects “placeholder” tool results by searching for the magic substring [APPROVAL_PENDING] inside function_result.result. In this framework, function_result.result is just arbitrary tool output (python/packages/core/agent_framework/_types.py:809-865), so using it as control-flow state leaks middleware implementation details into the generic content model and can misfire on legitimate tool output. That should be modeled as structured state instead of string matching. I found two design-level problems in the new security layer. The feature’s isolation model depends on module-global mutable state for per-invocation context, so it will behave correctly only as long as one tool invocation runs at a time in a process. In the existing async tool pipeline, that assumption does not hold: overlapping runs can clober each other’s security context, and separately configured agents can silently overwrite each other’s quarantine client. The main design problem is in the new DevUI policy-approval path: the mapper now emits policy_violation metadata, but the existing pending-approval state and submit flow still only preserve request_id and function_call, so legitimate UI approvals will drop the metadata while _executor now tries to rebuild security-sensitive state from client input. That is both incomplete and a step away from the existing server-validated approval model. I also saw one test change that weakens coverage by turning DevTools startup failures into skips rather than failures. That means the example does not actually exercise the cross-turn security behavior it claims to demonstrate, and it teaches readers the wrong integration pattern for any policy that depends on accumulated conversation state.

Flagged Issues

• python/samples/02-agents/security/email_security_example.py expects security context to carry across two independent agent.run() calls, but core agent code creates a new session when session= is omitted (_agents.py:1185-187, 1406-1408). Scenario 2 will not run in the tainted context from scenario 1. Reuse a single AgentSession or collapse into one run.

Suggestions

• Represent pending-approval tool results with explicit metadata or a dedicated content shape, then replace them by call_id using that structured marker instead of inspecting function_result.result text.
• Keep the quarantine chat client on SecureAgentConfig/middleware state and have quarantined_llm() obtain it from the current invocation context, preserving per-agent isolation.
• Reuse a single AgentSession in the email sample's CLI scenario so the second step actually demonstrates policy enforcement after untrusted content has entered the conversation.

---

Automated review by eavanvalkenburg's agents

[TaoChenOSU]

nit: is it possible to also discuss how these additional properties are consumed downstream?

[shrutitople]

Added a description on how the labels are used.

[TaoChenOSU]

nit: it's unclear from reading the ADR and this to understand how these work. Could we provide more details?

[shrutitople]

Included a small paragraph with the main purpose of automated hiding.

[TaoChenOSU]

Didn't we rename this to tool?

[shrutitople]

Correct. Removed.

[TaoChenOSU]

nit: Should the name be more explicit? For example, SecurityLabeledMessage? This mainly has two advantages:

1. Make it clear that this type contains the security label.
2. Reserve the LabeledMessage name in case we have some other labeled messages in the future.

[shrutitople]

Agreed.

[TaoChenOSU]

nit: should we make this param required?

[shrutitople]

I don't think it is necessary. There are clear defaults for each of them. The user and system prompt are trusted, the assistant message is trusted based on the initial trusted context and tool messages default to untrusted.

[TaoChenOSU]

I don't see this used anywhere. And why is this in this middleware that is a function middleware?

[shrutitople]

You are right. These were helper messages for the developer and are not used internally. Removed them for now as the developer does not need to explicitly label messages.

[TaoChenOSU]

I am slightly confused about this. If the result of the function is hidden, how do we expect the LLM/agent to proceed? And if we expect the LLM/agent to not proceed, why would the tool be given to the LLM/agent in the first place?

This may be related to the security boundary.

[shrutitople]

If the security label for the tool result is untrusted, then it is hidden inside a variable before adding it to the conversation history. This prevents the LLM's context from being tainted and exposed to potentially malicious data. For all the task, we do expect the LLM to proceed. For many operations, the LLM can simply pass the variable reference and the untrusted content will be exposed only inside the tool (Data-Independent tasks, Page 11). If the LLM has to look at the untrusted content to make decisions about the next tool call (Data-dependent tasks), then it can use the inspect_variable() tool call but this taints the context.

[TaoChenOSU]

Should the agent have tools for the middleware to take effect?

But again, if a tool cannot be trusted, why would it be given to the agent in the first place?

[shrutitople]

The implementation of the tool is always trusted, it is the content that the tool fetches could be untrusted for e.g., external emails, webcontent or anything where an adversary can potentially insert prompt injection instructions.

[TaoChenOSU]

Question: is this just shifting the risk to another client?

[shrutitople]

No, the quarantined_llm() as the name suggests does not have access to any tool calls. It can only perform data processing operations such as summarizing, parsing, extracting content. It takes untrusted content hidden inside variables as input, inspects it, operates on it and generates untrusted content which is again hidden inside variables as output. This ensures that the context of the planner LLM is not tainted. If the planner LLM absolutely needs to inspect the variable to decide the next tool call, then it uses the inspect_variable tool call which taints the context to untrusted. Any further tool call that is not allowed in an untrusted context will be either blocked or sent for human approval depending on the params. This is similar to the use of Dual LLM pattern https://www.agentic-patterns.com/patterns/dual-llm-pattern/

[TaoChenOSU]

Question: there are still no determinism and security guarantee as promised in the ADR? Prompt injection can reverse this instruction, or differently models may follow the instruction differently.

[shrutitople]

A model can interpret the instructions differently but it does not affect the security. The determinism and security guarantees are enforced by the security policies on each tool call. If the policy is violated (for e.g., because a model chose to use inspect_variable() for operating on untrusted data and therefore tainting the context) then the security falls back to the baseline of requesting a human approval with the attached message that this tool call is performed as a result of untrusted data being exposed to the model.

[github-actions]

Automated Code Review

Reviewers: 4 | Confidence: 90%

✗ Correctness

The main code changes in _tools.py improve approval-flow robustness (null-checks, call_id-based matching instead of positional indexing, empty message cleanup). The observability.py fix is correct and necessary. However, there is one concrete regression: the normal success path of _auto_invoke_function silently drops additional_properties propagation from the function call to the function result, while every other code path in the same function (no-middleware direct execution, MiddlewareTermination, Exception, and error paths) continues to propagate them. This inconsistency means the same function behaves differently depending on whether middleware is present and whether execution succeds or fails. The _parse_github_mcp_labels function in security.py contains a correctness bug: it uses lexicographic string comparison (field_conf.value > most_restrictive_confidentiality.value) to determine the most restrictive confidentiality label. Because "private" < "public" in Python string ordering (since 'r' < 'u'), PRIVATE confidentiality from GitHub MCP labels is incorrectly treated as less restrictive than PUBLIC, causing PRIVATE data to remain labeled PUBLIC. This occurs at two locations (lines 764 and 782). The rest of the codebase (e.g., combine_labels, check_confidentiality_allowed) correctly uses a priority dict for this comparison.

✗ Security Reliability

The _tools.py changes improve approval-flow robustness (call_id-based matching instead of positional, null-safety on approved_function_call) and add security middleware integration points. Two reliability concerns: (1) the normal middleware success path silently drops additional_properties that error/termination paths still propagate, creating inconsistent metadata on function results; (2) placeholder replacement uses a magic string sentinel '[APPROVAL_PENDING]' in tool-result text, which is fragile—a structured Content attribute would be safer. The observability fix for non-string finish_reason is correct and needed. Two security-relevant bugs found in the new security.py module: (1) The GitHub MCP label parser uses lexicographic string comparison for confidentiality levels, causing PRIVATE content to be misclassified as PUBLIC because "private" < "public" lexicographically. This undermines confidentiality protections for GitHub MCP-sourced data. (2) Thread-local storage (threading.local()) is used to pass the middleware instance to async security tools, but threading.local() does not isolate concurrent asyncio coroutines on the same thread, risking cross-coroutine middleware leakage. The production code changes add policy-violation metadata plumbing through the approval flow. The _executor.py change spreads untrusted client-supplied dict data (**policy_violation_data) into additional_properties, which allows a malicious client to inject arbitrary keys or override the policy_violation: True flag. The _mapper.py changes safely extract specific keys. Test changes and foundry test fix are fine.

✗ Test Coverage

The PR introduces significant new logic in _tools.py — placeholder [APPROVAL_PENDING] replacement, empty-message cleanup, function_approval_request passthrough from middleware, and call_id null validation — but none of these new code paths have corresponding unit tests in test_function_invocation_logic.py. The observability isinstance(finish_reason, str) guard also lacks a test for non-string finish_reason values. While test_security.py is listed as a new file, it does not exist on disk and presumably appears in a later chunk; the gaps noted here are in the core _tools.py approval flow, which should be tested independently of the security module. The security module contains a correctness bug in _parse_github_mcp_labels where confidentiality levels are compared via string ordering ("private" < "public" alphabetically) instead of the intended hierarchy (PUBLIC < PRIVATE < USER_IDENTITY), causing PRIVATE data to not be recognized as more restrictive than PUBLIC. No test file for the security module exists on disk — test_security.py is listed as a changed file but appears in another chunk, so its coverage cannot be verified here. A test specifically covering _parse_github_mcp_labels with PRIVATE fields would catch this bug. The new test_security.py file is extensive (~2578 lines) and covers the major security subsystems well: label creation/serialization, combine_labels, ContentVariableStore, middleware label tracking, policy enforcement, auto-hiding, tiered label propagation, confidentiality checks, and quarantined LM. Three test coverage gaps stand out: (1) the devui _executor.py and _mapper.py changes add policy_violation extraction and mapping logic with zero corresponding tests, (2) test_audit_log_recording only asserts the initial empty state and never triggers a violation to verify recording, and (3) the new _replace_approval_contents_with_results tests in test_function_invocation_logic.py are well-structured and cover the key call-id-matching fix thoroughly.

✗ Design Approach

That is a contract regression rather than a local bug: the framework already treats additional_properties as the extensibility channel on Content, and the rest of _auto_invoke_function still preserves it. With middleware enabled, result metadata now disappears only on the success path, which is especially risky for any higher-level features that depend on result labels or other propagated metadata. First, it stores per-invocation middleware state in a thread-local even though the core tool loop executes function calls concurrently with asyncio.gather, so concurrent tool calls can read or clear each other’s security context. Second, the GitHub MCP label parser derives confidentiality ordering from enum string values instead of the explicit confidentiality lattice used elsewhere, which can silently downgrade private content during label propagation. I also found the new memory-test skip broad enough to hide real browser/DevTools startup regressions instead of isolating a known unsupported environment. I found two design-level problems in the new security samples. First, the email demo assumes the security context stays tainted across two separate agent.run(...) calls, but the core agent only preserves provider/history state when you reuse an explicit session, so the second step does not reliably demonstrate the policy boundary it claims to show. Second, both samples hardcode gpt-4o-mini as the quarantine Foundry model even though FoundryChatClient treats model as a deployment name, which makes the samples depend on a very specific project setup instead of a configurable quarantine deployment.

Flagged Issues

• The [APPROVAL_PENDING] placeholder replacement logic in _replace_approval_contents_with_results (lines 1872–1882 of _tools.py) is entirely new behavior that modifies message contents in-place, but has zero test coverage across all test files.

Suggestions

• Narrow the memory regression test skip — converting the generic DevTools timeout into pytest.skip() masks browser-launch and remote-debugging regressions this test is supposed to catch. Skip only for a narrowly identified unsupported-environment condition.
• Reuse an explicit session in the email sample (e.g., session = agent.create_session() passed to both agent.run() calls) so the second step genuinely demonstrates cross-turn policy enforcement.
• Make the quarantine deployment configurable (e.g., FOUNDRY_QUARANTINE_MODEL env var with fallback to FOUNDRY_MODEL) and update documentation to reference deployment names rather than a literal gpt-4o-mini string.
• Add unit tests for quarantined_llm both with and without a quarantine client configured, verifying the response is labeled UNTRUSTED and tool_choice='none' is passed.

---

Automated review by moonbox3's agents

[github-actions]

Missing test coverage: The second pass that removes messages with empty contents lists is new logic. No existing test verifies that messages left empty after content removal are pruned from the messages list.