Follow me on LinkedIn for useful Databricks projects and tips. Training materials are also available on my website: DataEngineer.wiki
Looking for more practice?
Check out my other hands-on Databricks labs covering Delta Live Tables, table optimization, and more at dataengineer.wiki/projects.
Looking for more info on passing the Databricks Data Engineer Associate Certification exam?
Check out helpful resources including YouTube videos and official Databricks courses at dataengineer.wiki/certifications/data-engineer-associate.
This lab provides hands-on practice to prepare for the Databricks Certified Data Engineer Associate exam. You will build a production-grade, end-to-end data pipeline using real-world scenarios and datasets. The exercises are designed to be completed within the Databricks Free Community Edition, allowing you to develop practical skills without any cost.
The lab covers the entire data engineering lifecycle, including:
- Ingesting raw data from various sources using Auto Loader and COPY INTO.
- Implementing the Medallion Architecture (Bronze, Silver, Gold layers).
- Performing data transformations, quality checks, and implementing SCD Type 2.
- Orchestrating workflows with Databricks Jobs and multi-task dependencies.
- Managing data governance using Unity Catalog with role-based access control.
This lab is structured to cover the key topics outlined in the official Databricks Data Engineer Associate exam info. By completing the notebooks, you will gain practical experience in the following areas:
- What you'll practice: Enabling features that simplify data layout decisions, understanding the value of the Data Intelligence Platform, and identifying the applicable compute for specific use cases.
- What you'll practice: Leveraging Notebooks functionality, working with Auto Loader from various sources (JSON, CSV), using COPY INTO for batch incremental loads, and handling schema evolution.
- What you'll practice: Implementing the three layers of the Medallion Architecture (Bronze, Silver, Gold), performing data quality transformations, implementing Slowly Changing Dimensions (SCD Type 2), and computing complex aggregations with PySpark window functions.
- What you'll practice: Creating and configuring Databricks Jobs, implementing multi-task workflows with dependencies, using job parameters with widgets, implementing error handling and retry logic, and using serverless compute.
- What you'll practice: Understanding Unity Catalog's three-level namespace (catalog.schema.table), creating and managing catalogs, schemas, and volumes, creating managed tables, implementing access control with GRANT and REVOKE statements, and data quality validation patterns.
This lab focuses on core data engineering workflows. If you want to practice additional exam topics such as Delta Live Tables (DLT) pipelines or Delta table optimization techniques, check out my other hands-on labs at dataengineer.wiki/projects.
Register for a "Virtual Learning Festival", complete the required courses in the timeline provided to automatically receive 50% off the certification.
Look for upcoming Databricks Festival here - https://community.databricks.com/t5/events/eb-p/databricks-community-events
Follow these three simple steps to begin:
- Go to databricks.com/learn/free-edition.
- Sign up for the Free Edition. This gives you access to all the necessary tools, including serverless compute and Unity Catalog.
- In your Databricks workspace, navigate to Workspace > Repos.
- Click Add Repo.
- For the Git repository URL, paste:
https://github.com/jrlasak/databricks_data_engineer_associate_cert_prep. - Click Create Repo.
- Once the repo is cloned, navigate to the
notebooks/directory. - Open and run the
00_Setup_Environment.pynotebook to configure your workspace. - Proceed through the notebooks in numerical order, starting with
01_Environment_Setup_Unity_Catalog.py.
Each notebook contains exercises marked with TODO and corresponding solutions for you to check your work. Good luck!