Skip to content

feat: Azure Data Engineering tutorials collection (10 modules)#2

Draft
Copilot wants to merge 2 commits intomainfrom
copilot/azure-data-engineering-collection
Draft

feat: Azure Data Engineering tutorials collection (10 modules)#2
Copilot wants to merge 2 commits intomainfrom
copilot/azure-data-engineering-collection

Conversation

Copy link
Copy Markdown

Copilot AI commented Apr 14, 2026

Adds a new azure-data-engineering/ top-level directory covering the full breadth of high-demand Azure Data Engineering topics, each as a self-contained module with a README, runnable code, and sample data where applicable.

Modules

Module Artifact
ADF adf_pipeline_simulator.py — watermark-based Copy Activity simulation
Databricks databricks_pyspark_demo.ipynb — PySpark transforms, window functions, Delta write
Synapse Analytics synapse_analytics_demo.py — CETAS, incremental load, partition elimination (DuckDB local sim)
ADLS Gen2 adls_gen2_demo.py — HNS operations, ACLs, Medallion structure via Azure SDK
Delta Lake delta_lake_demo.ipynb — CRUD, MERGE/upsert, time travel, schema evolution, transaction log
SQL & PySpark sql_pyspark_demo.ipynb — joins, window functions, pivot/unpivot, perf tips
Python / Azure SDK azure_sdk_demo.py — Blob Storage, Key Vault, ADF pipeline trigger + poll
Snowflake snowflake_demo.py — COPY INTO, Snowpark, streams/tasks, time travel, zero-copy clone
Microsoft Fabric fabric_lakehouse_demo.py — Fabric REST API, OneLake upload, SQL Analytics Endpoint
Scenario Q&A scenario_pipeline_designer.py — 5 real-world pipeline design scenarios with model answers

Design notes

  • All Azure-dependent scripts detect missing credentials and fall back to dry-run mode, printing operations rather than executing them — no Azure account required to follow along.
  • Root README.md updated with a new Azure Data Engineering section linking all modules.
  • snowflake/sample_orders.csv includes the total column to match the target table schema used in COPY INTO.
  • Removed secret.value reference from Key Vault log call to avoid clear-text logging of sensitive data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants