0.7 alpha 2#258
Open
Chenglong-MS wants to merge 384 commits intomainfrom
Open
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.5.4 to 6.5.5. - [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst) - [Commits](tornadoweb/tornado@v6.5.4...v6.5.5) --- updated-dependencies: - dependency-name: tornado dependency-version: 6.5.5 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
Bump cryptography from 46.0.4 to 46.0.6
Bump pygments from 2.19.2 to 2.20.0
Bump aiohttp from 3.13.3 to 3.13.4
Bump litellm from 1.81.6 to 1.83.0
Bump tornado from 6.5.4 to 6.5.5
…vider model, transitioning from a multi-provider chain to a single provider with anonymous fallback. Updated sections on protocol support and plugin design patterns for improved clarity and structure.
- 移除1-sso-plugin-architecture.md中重复的插件接口定义，改为引用1-data-source-plugin-architecture.md - 新增2-external-dataloader-enhancements.md详细说明外部数据加载器的三项改进方案： 1. 数据库元数据拉取（P0） 2. SSO Token透传（P1） 3. 凭证持久化（P2） - 明确各数据库实现细节和优先级规划
添加详细的开发路线图文档，包含SSO认证、数据源插件框架、凭证保险箱等功能的实施计划与测试策略
补充 Superset 集成代码迁移说明，调整步骤编号，并添加文档交付要求
补充数据溯源描述的设计决策和实现方式，使用模板拼接而非AI生成来保证准确性和可刷新性。描述内容包括来源、筛选条件、时间范围等，并自动存储到loader_metadata中供前端和AI使用。
…ching issues Ensure table key values are validated when building catalog tree Propagate table_key to metadata for frontend use Fix frontend annotation data fetching path
Add SandboxSession class to support sharing Python namespace across multiple explore/execute_python calls within the same agent turn. Main changes include: 1. Extend worker protocol to support 4-tuple messages and __clear_ns__ directive 2. Add SandboxSession context manager encapsulating persistence logic 3. DataAgent and DataLoadingAgent integrate session management 4. Support cross-turn namespace save/restore (via parquet+manifest) 5. Add related test cases and security checks Resolve excessive tool call issues, improve multi-step data analysis efficiency.
Feature/plugin architecture
|
|
||
| The response body must not include endpoint-specific top-level fields. | ||
| """ | ||
| return jsonify({"status": "success", "data": data}), status_code |
Check warning
Code scanning / CodeQL
Information exposure through an exception Medium
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 3 days ago
General fix: never include raw exception text ({e}, {exc}) in client-facing response fields. Log full exceptions server-side (logger.exception(...)) and return stable generic messages (optionally with safe error codes). This preserves functionality while preventing information leakage.
Best targeted fix here:
- File:
py-src/data_formulator/sandbox/docker_sandbox.py- In both
except Exception as excblocks, keep logging but replacedetailargument currently built from exception text with a generic diagnostic string.
- In both
- File:
py-src/data_formulator/sandbox/local_sandbox.py- Import
sanitize_error_message. - In the two exception handlers returning
"content"with{e}, replace with generic fixed user messages. - Add server-side logging via
logger.exception(...). - Optionally keep a sanitized diagnostic field to preserve troubleshooting value without exposing raw internals.
- Import
This removes the source of tainted exception data for all listed variants and keeps outward API behavior (error status/content shape) intact.
Suggested changeset
2
py-src/data_formulator/sandbox/local_sandbox.py
Outside changed files
| @@ -17,6 +17,7 @@ | ||
|
|
||
| import pandas as pd | ||
|
|
||
| from data_formulator.security.sanitize import sanitize_error_message | ||
| from .base import Sandbox | ||
|
|
||
| logger = logging.getLogger(__name__) | ||
| @@ -535,9 +536,13 @@ | ||
| } | ||
|
|
||
| except Exception as e: | ||
| logger.exception("[LocalSandbox] Error during execution setup") | ||
| return { | ||
| "status": "error", | ||
| "content": f"Error during execution setup: {type(e).__name__} - {e}", | ||
| "content": "Error during execution setup.", | ||
| "diagnostics": { | ||
| "safe_detail": sanitize_error_message(f"{type(e).__name__}: {e}") | ||
| }, | ||
| } | ||
|
|
||
| # ------------------------------------------------------------------ | ||
| @@ -569,7 +572,14 @@ | ||
| _worker_pool.release(proc, conn) | ||
| return result | ||
| except Exception as e: | ||
| logger.exception("[LocalSandbox] Worker communication failed") | ||
| _worker_pool.discard(proc, conn) | ||
| return {"status": "error", "content": f"Error: worker communication failed - {e}"} | ||
| return { | ||
| "status": "error", | ||
| "content": "Error: worker communication failed.", | ||
| "diagnostics": { | ||
| "safe_detail": sanitize_error_message(f"{type(e).__name__}: {e}") | ||
| }, | ||
| } | ||
|
|
||
|
|
py-src/data_formulator/sandbox/docker_sandbox.py
Outside changed files
| @@ -193,7 +193,7 @@ | ||
| self._cleanup(tmpdir) | ||
| return _safe_error_response( | ||
| "Failed to start Docker container.", | ||
| f"{type(exc).__name__}: {exc}", | ||
| "Sandbox startup failed", | ||
| ) | ||
|
|
||
| stdout = proc.stdout or "" | ||
| @@ -243,7 +243,7 @@ | ||
| self._cleanup(tmpdir) | ||
| return _safe_error_response( | ||
| "Failed to read output parquet.", | ||
| f"{type(exc).__name__}: {exc}", | ||
| "Sandbox output read failed", | ||
| ) | ||
|
|
||
| self._cleanup(tmpdir) |
Copilot is powered by AI and may make mistakes. Always verify output.
Collaborator
There was a problem hiding this comment.
Won't fix for now. The flagged exception text is used as part of the sandbox execution feedback loop, not just as a generic user-facing error. Sandbox failures are fed back into the agent so the model can repair generated Python code based on concrete diagnostics such as NameError, KeyError, SyntaxError, missing columns, missing output variables, or DataFrame type mismatches.
Replacing these messages with a generic error would likely reduce repair quality and could break existing agent workflows where the next model turn depends on the specific execution failure. The current behavior is therefore intentional for code-execution diagnostics.
We acknowledge that raw host/sandbox infrastructure exceptions should not be exposed broadly. A future targeted fix should preserve sanitized user-code execution errors for model repair while separately replacing infrastructure-level failures, such as worker communication or Docker startup failures, with stable generic messages and server-side logging. @Chenglong-MS
- Use ConfinedDir to restrict file operation scope, prevent symlink escape - Strictly validate origin and safely render JWT templates in SSO bridge - Add test cases to verify security protection measures
Remove function return value and parameter type annotations to support older Python runtime environments. These modifications primarily affect origin validation functionality related to SSO bridging.
Feature/plugin architecture
…ring - Add safe path check function to prevent open redirects - Improve error message handling, filter sensitive information and stack traces - Implement unified secure error responses in Docker sandbox and agents - Add test cases to verify security filtering functionality
…rning functionality Add sanitize_error_message feature in error handling module to securely sanitize all error details returned to clients Add streaming warning handling mechanism, including stream_warning_event and collect_stream_warning/flush_stream_warnings utilities Update development documentation, supplement streaming warning specifications and usage instructions
fix(oidc): use issuer returned by discovery endpoint to update configuration docs: update OIDC security guide, add GitHub private email and state validation instructions feat(github): add state parameter validation and private email retrieval functionality test: add contract tests for mainstream SSO providers
refactor(测试): 优化BigQuery模拟器探针逻辑避免阻塞 docs(测试): 更新README文档说明新的测试服务管理方式 fix(测试): 移除MySQL和PostgreSQL测试中不必要的row_count检查 style(测试): 格式化Superset测试代码并更新API端点
Feature/plugin architecture
…ensitive query parameters refactor(frontend): extract rule pre-content processing logic into independent module docs: update documentation for error handling and log desensitization test: add unit tests for log desensitization and rule pre-content
fix(security): enhance log desensitization functionality to support s…
Add test cases to verify automatic log rotation by date functionality Refactor ReasoningLogger class, extract log file path related properties as instance variables Add _ensure_fd_for_today method to handle log file switching when date changes
…ing document Add document 15.3 with detailed planning for Agent knowledge injection matrix and search tool implementation Update document 15 table format and add link to document 15.3
1. 重构元数据同步流程，支持从缓存读取富元数据 2. 在数据摘要中自动注入目录元数据（表/列描述、标签等） 3. 前端展示合并后的元数据状态和用户标注提示 4. 新增测试验证元数据同步和展示逻辑 - 目录缓存支持两种写入模式（replace/seed_if_missing） - 搜索工具返回source_id/table_key用于元数据读取 - 优化数据加载时的行数截断提示 - 统一元数据状态枚举和前端多语言支持
…on field support Add support for verbose_name and expression fields in Agent context, while preserving dual-source descriptions from source_description and user_description
Feature/plugin architecture
Add _build_sort_clause static method for building ORDER BY clauses Modify fetch_data_as_arrow method to support sorting options Add related unit tests to verify sorting functionality
… security design document Add ISSUE-005 design document, detailing the current status of sorting capabilities and column name concatenation security issues in data loaders, root cause analysis, fix plan, and testing strategy
Feature/plugin architecture
… library Merge skills and experiences directories into a unified library directory, keeping rules independent. Main changes include: - Modify constant definitions and initialization logic in knowledge/store.py - Update category references in API routes and Agent tools - Adjust frontend type definitions and state management - Optimize ExperienceDistillAgent prompts to distill general methodologies - Update related test cases
…periences - Merge original three directories rules/skills/experiences into rules/experiences - KnowledgeStore.search() automatically skips rules with alwaysApply=true - Update ExperienceDistillAgent to distill general methodologies rather than specific cases - Frontend simplified from three tabs to Rules/Experiences two sections - Injection text uses semantic tags [knowledge]/[rule] instead of directory names
…ience distillation Add timeout parameter configuration in API, backend routes, and agent, frontend sets request timeout based on configuration
…gorithm Refactor knowledge base rule injection logic, unify duplicate code into KnowledgeStore.format_rules_block() method, support preloading data to avoid secondary disk reads. Improve search algorithm to tokenization + multi-field weighted matching, support Chinese-English mixed query splitting and table name tag bonus. Update related documentation and test cases. - Add format_rules_block() and load_always_apply_rules() methods - Implement _tokenize_query() supporting Chinese-English mixed tokenization - Improve _match_score() weighted algorithm and table name tag bonus - Update DataLoadingAgent to use last user message as search query - Refactor rule injection code in 6 Agents - Improve test coverage and development documentation
- Update Superset SSO configuration example, add DF Token Exchange endpoint - Supplement SSO token exchange mode documentation, including flow, deployment steps, and security notes - Mark user metadata feature as completed, abandon imported table editing approach - Streamline knowledge system documentation, archive completed content - Update user isolation design document, record implemented portions - Adjust knowledge injection planning document, highlight core conclusions and implemented portions
Add Chinese and English translations for Agent logs, including status messages and expand/collapse functionality Implement long message folding and expanding functionality, optimize message display experience Enhance Agent step display, support icon differentiation for error, warning, and info states Fix JSON parsing issues when weak models call tools, add validation logic
Feature/plugin architecture
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Summary
Agents & AI Pipeline
agent_py_data_rec,agent_sql_data_rec,agent_py_data_transform,agent_sql_data_transform,agent_concept_derive,agent_py_concept_derive,agent_data_clean, andagent_explorationinto three unified agents:data_agent.py,agent_data_rec.py, andagent_data_transform.pysemantic_types.pybackend module and full frontend type registry (src/lib/agents-chart/core/type-registry.ts,field-semantics.ts,semantic-types.ts) with domain shape inference, tick constraints, zero-baseline classification, and snap-to-bound heuristicsagent_chart_insight.pyfor AI-generated chart takeawaysagent_language.pyfor i18n-aware promptsagent_diagnostics.pywith unified diagnostic information builder for better error reportingVisualization
src/lib/agents-chart/, 120 files, ~44K lines) with multi-backend support for Vega-Lite, ECharts, Chart.js, and GoFish — includes template system, semantic-aware axis/domain/tick handling, color decisions, layout computation, faceting, and overflow filteringChartGallery.tsxwith expanded chart type support including pie, US map, world map, bump, candlestick, density, lollipop, pyramid, radar, rose, streamgraph, strip plot, waterfall, and moreChartRenderService.tsxreplacing static SVG rendering withvega-embedfor interactive chartsSimpleChartRecBox.tsxandchartRecommendation.tsfor improved chart suggestion workflowScoretype with small domain spans (e.g., [0,1]) no longer forces integer-only ticks, preserving intermediate decimal ticksData Thread & Workflow
DataThread.tsxrewrite, newDataThreadCards.tsx,InteractionEntryCard.tsx)useFormulateData.tsconsolidating data derivation logicTiptapReportEditor.tsx) with richer editing supportData Loading & Management
UnifiedDataUploadDialog.tsxreplacing the old table selection view — supports file upload, URL, paste, database, and sample datasets in a single dialog with loading state indicatorsMultiTablePreview.tsxfor previewing multiple tables before loadingtableThunks.tshandling all data source types with server-side workspace storageuseDataRefresh.tsxwith auto-refresh, stream data sources, andRefreshDataDialog.tsx#rowId) viaROW_NUMBER()in DuckDB and pandas paths, preserving original row positions after sortData Loaders (Database Plugins)
Datalake / Workspace Backend
datalake/package withworkspace.py,azure_blob_workspace.py,cached_azure_blob_workspace.py,file_manager.py,metadata.py,cache_manager.py,parquet_utils.py, andtable_names.pyworkspace_factory.pyfor configuration-driven workspace initializationsession_routes.pyfor session-level API endpointsSecurity
code_signing.pyfor generated code integrity verificationauth.pyfor authentication handlingurl_allowlist.pyfor URL validationsanitize.pyto prevent leaking sensitive info in error messagessandbox/package withlocal_sandbox.py,docker_sandbox.py,not_a_sandbox.py, andDockerfile.sandboxreplacing the oldpy_sandbox.pyidentity.tswith browser-based identity for multi-user supportInternationalization (i18n)
react-i18nextwith English and Chinese locale files across 7 namespaces (common, chart, encoding, messages, model, navigation, upload)TRANSLATION_GUIDE.mdfor contributorsUI & Design System
tokens.tswith centralized color, spacing, shadow, transition, and radius tokensDataFormulator.tsxandApp.tsxwithTopNavButton,AppShellnavigation, and model management UIEncodingShelfCard.tsxandEncodingShelfThread.tsxConceptCard.tsx,ConceptShelf.tsx,DerivedDataDialog.tsxModel Management
model_registry.pyfor managing model configurations server-sideModelSelectionDialog.tsxwith multi-model supportInfrastructure & DevOps
Dockerfile,docker-compose.yml,docker-compose.test.ymlwith volume permissions and sandbox user handling.devcontainer/devcontainer.jsonuv.lock, updatedpyproject.tomlandrequirements.txtTesting
vitest.config.ts,pytest.ini,conftest.py, frontend setup, andtest_plan.md