Build practical programming and technology skills

Data Engineering Fundamentals

Design, build, test, secure, observe, and operate reliable batch and streaming-oriented data pipelines, storage layers, models, and workflows.

View curriculum

ProfessionalDifficulty

50 hoursEstimated time

8 modules32 lessons

5 guided labsKnowledge check

Plan your learning

Path Brief

Skills you will practice

Data Ai · Data Engineering Fundamentals
Data Ai · Data Engineering Architecture
Data Ai · Data Formats And Storage
Data Ai · Batch Ingestion And Transformation
Data Ai · Data Modeling And Warehousing
Data Ai · Workflow Orchestration

Prerequisites

Python Fundamentals
Sql Relational Databases
Git Github

Tools

A current web browser
A code editor or terminal appropriate to the path
A safe local or virtual lab environment

Before you begin

What you need

Python Fundamentals
Sql Relational Databases
Git Github

Step-by-step curriculum

Learning modules

Complete lessons in order or open the module that matches your current goal. Your lesson progress is saved on this device.

01Data Engineering ArchitectureBuild practical skill in data engineering architecture and connect it to the outcomes of Data Engineering Fundamentals.4 lessons · 1 lab

Lesson 1

Data Engineering Architecture: Core Concepts for Data Engineering Fundamentals

Understand the concepts, decisions, and vocabulary you need before applying Data Engineering Architecture in Data Engineering Fundamentals.

25 min

Lesson 2

Set Up and Explore Data Engineering Architecture in Data Engineering Fundamentals

Prepare the tools, data, project state, or test environment needed to explore Data Engineering Architecture safely and repeatably.

25 min

Lesson 3

Build a Practical Data Engineering Architecture Example in Data Engineering Fundamentals

Build a small batch pipeline that reads a source file, validates schema, transforms rows, writes output, and can be rerun without duplication.

30 min

Lesson 4

Debug Common Data Engineering Architecture Problems in Data Engineering Fundamentals

Recognize common failure modes in Data Engineering Architecture, use the relevant diagnostics, and verify the correction.

25 min

Guided lab · 55 min · Safe

Guided Lab: Data Engineering Architecture in Data Engineering Fundamentals

Data Engineering Fundamentals — Data Engineering Architecture lab: Build a small batch pipeline that reads a source file, validates schema, transforms rows, writes output, and can be rerun without duplication.

Not started

02Data Formats and StorageBuild practical skill in data formats and storage and connect it to the outcomes of Data Engineering Fundamentals.4 lessons · 1 lab

Lesson 1

Data Formats and Storage: Core Concepts for Data Engineering Fundamentals

Understand the concepts, decisions, and vocabulary you need before applying Data Formats and Storage in Data Engineering Fundamentals.

25 min

Lesson 2

Set Up and Explore Data Formats and Storage in Data Engineering Fundamentals

Prepare the tools, data, project state, or test environment needed to explore Data Formats and Storage safely and repeatably.

25 min

Lesson 3

Build a Practical Data Formats and Storage Example in Data Engineering Fundamentals

Store a small record, restart or reload the application, and prove the state can be read back correctly.

30 min

Lesson 4

Debug Common Data Formats and Storage Problems in Data Engineering Fundamentals

Recognize common failure modes in Data Formats and Storage, use the relevant diagnostics, and verify the correction.

25 min

Guided lab · 55 min · Moderate

Guided Lab: Data Formats and Storage in Data Engineering Fundamentals

Data Engineering Fundamentals — Data Formats and Storage lab: Store a small record, restart or reload the application, and prove the state can be read back correctly.

Not started

03Batch Ingestion and TransformationBuild practical skill in batch ingestion and transformation and connect it to the outcomes of Data Engineering Fundamentals.4 lessons · 1 lab

Lesson 1

Batch Ingestion and Transformation: Core Concepts for Data Engineering Fundamentals

Understand the concepts, decisions, and vocabulary you need before applying Batch Ingestion and Transformation in Data Engineering Fundamentals.

25 min

Lesson 2

Set Up and Explore Batch Ingestion and Transformation in Data Engineering Fundamentals

Prepare the tools, data, project state, or test environment needed to explore Batch Ingestion and Transformation safely and repeatably.

25 min

Lesson 3

Build a Practical Batch Ingestion and Transformation Example in Data Engineering Fundamentals

Build a small batch pipeline that reads a source file, validates schema, transforms rows, writes output, and can be rerun without duplication.

30 min

Lesson 4

Debug Common Batch Ingestion and Transformation Problems in Data Engineering Fundamentals

Recognize common failure modes in Batch Ingestion and Transformation, use the relevant diagnostics, and verify the correction.

25 min

Guided lab · 55 min · Moderate

Guided Lab: Batch Ingestion and Transformation in Data Engineering Fundamentals

Data Engineering Fundamentals — Batch Ingestion and Transformation lab: Build a small batch pipeline that reads a source file, validates schema, transforms rows, writes output, and can be rerun without duplication.

Not started

04Data Modeling and WarehousingBuild practical skill in data modeling and warehousing and connect it to the outcomes of Data Engineering Fundamentals.4 lessons · 1 lab

Lesson 1

Data Modeling and Warehousing: Core Concepts for Data Engineering Fundamentals

Understand the concepts, decisions, and vocabulary you need before applying Data Modeling and Warehousing in Data Engineering Fundamentals.

25 min

Lesson 2

Set Up and Explore Data Modeling and Warehousing in Data Engineering Fundamentals

Prepare the tools, data, project state, or test environment needed to explore Data Modeling and Warehousing safely and repeatably.

25 min

Lesson 3

Build a Practical Data Modeling and Warehousing Example in Data Engineering Fundamentals

Design two related tables or models, add constraints, and create a migration that can be applied to a test database.

30 min

Lesson 4

Debug Common Data Modeling and Warehousing Problems in Data Engineering Fundamentals

Recognize common failure modes in Data Modeling and Warehousing, use the relevant diagnostics, and verify the correction.

25 min

Guided lab · 55 min · Moderate

Guided Lab: Data Modeling and Warehousing in Data Engineering Fundamentals

Data Engineering Fundamentals — Data Modeling and Warehousing lab: Design two related tables or models, add constraints, and create a migration that can be applied to a test database.

Not started

05Workflow OrchestrationBuild practical skill in workflow orchestration and connect it to the outcomes of Data Engineering Fundamentals.4 lessons · 1 lab

Lesson 1

Workflow Orchestration: Core Concepts for Data Engineering Fundamentals

Understand the concepts, decisions, and vocabulary you need before applying Workflow Orchestration in Data Engineering Fundamentals.

25 min

Lesson 2

Set Up and Explore Workflow Orchestration in Data Engineering Fundamentals

Prepare the tools, data, project state, or test environment needed to explore Workflow Orchestration safely and repeatably.

25 min

Lesson 3

Build a Practical Workflow Orchestration Example in Data Engineering Fundamentals

Build a small batch pipeline that reads a source file, validates schema, transforms rows, writes output, and can be rerun without duplication.

30 min

Lesson 4

Debug Common Workflow Orchestration Problems in Data Engineering Fundamentals

Recognize common failure modes in Workflow Orchestration, use the relevant diagnostics, and verify the correction.

25 min

Guided lab · 55 min · Moderate

Guided Lab: Workflow Orchestration in Data Engineering Fundamentals

Data Engineering Fundamentals — Workflow Orchestration lab: Build a small batch pipeline that reads a source file, validates schema, transforms rows, writes output, and can be rerun without duplication.

Not started

06Data Quality and ObservabilityBuild practical skill in data quality and observability and connect it to the outcomes of Data Engineering Fundamentals.4 lessons

Lesson 1

Data Quality and Observability: Core Concepts for Data Engineering Fundamentals

Understand the concepts, decisions, and vocabulary you need before applying Data Quality and Observability in Data Engineering Fundamentals.

25 min

Lesson 2

Set Up and Explore Data Quality and Observability in Data Engineering Fundamentals

Prepare the tools, data, project state, or test environment needed to explore Data Quality and Observability safely and repeatably.

25 min

Lesson 3

Build a Practical Data Quality and Observability Example in Data Engineering Fundamentals

Add one log, metric, and health signal to a small service and use them to diagnose a controlled failure.

30 min

Lesson 4

Debug Common Data Quality and Observability Problems in Data Engineering Fundamentals

Recognize common failure modes in Data Quality and Observability, use the relevant diagnostics, and verify the correction.

25 min

07Streaming ConceptsBuild practical skill in streaming concepts and connect it to the outcomes of Data Engineering Fundamentals.4 lessons

Lesson 1

Streaming Concepts: Core Concepts for Data Engineering Fundamentals

Understand the concepts, decisions, and vocabulary you need before applying Streaming Concepts in Data Engineering Fundamentals.

25 min

Lesson 2

Set Up and Explore Streaming Concepts in Data Engineering Fundamentals

Prepare the tools, data, project state, or test environment needed to explore Streaming Concepts safely and repeatably.

25 min

Lesson 3

Build a Practical Streaming Concepts Example in Data Engineering Fundamentals

Build a small batch pipeline that reads a source file, validates schema, transforms rows, writes output, and can be rerun without duplication.

30 min

Lesson 4

Debug Common Streaming Concepts Problems in Data Engineering Fundamentals

Recognize common failure modes in Streaming Concepts, use the relevant diagnostics, and verify the correction.

25 min

08Security, Cost, and OperationsBuild practical skill in security, cost, and operations and connect it to the outcomes of Data Engineering Fundamentals.4 lessons

Lesson 1

Security, Cost, and Operations: Core Concepts for Data Engineering Fundamentals

Understand the concepts, decisions, and vocabulary you need before applying Security, Cost, and Operations in Data Engineering Fundamentals.

25 min

Lesson 2

Set Up and Explore Security, Cost, and Operations in Data Engineering Fundamentals

Prepare the tools, data, project state, or test environment needed to explore Security, Cost, and Operations safely and repeatably.

25 min

Lesson 3

Build a Practical Security, Cost, and Operations Example in Data Engineering Fundamentals

Review a batch pipeline and add one least-privilege access rule, one data-quality/operability signal, and one cost or retention control; simulate a failed run and recovery.

30 min

Lesson 4

Debug Common Security, Cost, and Operations Problems in Data Engineering Fundamentals

Recognize common failure modes in Security, Cost, and Operations, use the relevant diagnostics, and verify the correction.

25 min

Portfolio integration

Capstone Project

Build an Observable Analytical Data Pipeline

Combine the path skills in a portfolio-ready project with clear functional requirements, tests, verification evidence, and operating notes.

GuidedIndependentPortfolio

Acceptance criteria

The core workflow works from start to finish.
Invalid input or failure states are handled clearly.
The result is tested and documented.
A reviewer can reproduce the setup and verification.

Continue in Programming

Related Programming

Skill assessment

Data Engineering Fundamentals

Path Brief

Skills you will practice

Prerequisites

Tools

What you need

Learning modules

Data Engineering Architecture: Core Concepts for Data Engineering Fundamentals

Set Up and Explore Data Engineering Architecture in Data Engineering Fundamentals

Build a Practical Data Engineering Architecture Example in Data Engineering Fundamentals

Debug Common Data Engineering Architecture Problems in Data Engineering Fundamentals

Guided Lab: Data Engineering Architecture in Data Engineering Fundamentals

Data Formats and Storage: Core Concepts for Data Engineering Fundamentals

Set Up and Explore Data Formats and Storage in Data Engineering Fundamentals

Build a Practical Data Formats and Storage Example in Data Engineering Fundamentals

Debug Common Data Formats and Storage Problems in Data Engineering Fundamentals

Guided Lab: Data Formats and Storage in Data Engineering Fundamentals

Batch Ingestion and Transformation: Core Concepts for Data Engineering Fundamentals

Set Up and Explore Batch Ingestion and Transformation in Data Engineering Fundamentals

Build a Practical Batch Ingestion and Transformation Example in Data Engineering Fundamentals

Debug Common Batch Ingestion and Transformation Problems in Data Engineering Fundamentals

Guided Lab: Batch Ingestion and Transformation in Data Engineering Fundamentals

Data Modeling and Warehousing: Core Concepts for Data Engineering Fundamentals

Set Up and Explore Data Modeling and Warehousing in Data Engineering Fundamentals

Build a Practical Data Modeling and Warehousing Example in Data Engineering Fundamentals

Debug Common Data Modeling and Warehousing Problems in Data Engineering Fundamentals

Guided Lab: Data Modeling and Warehousing in Data Engineering Fundamentals

Workflow Orchestration: Core Concepts for Data Engineering Fundamentals

Set Up and Explore Workflow Orchestration in Data Engineering Fundamentals

Build a Practical Workflow Orchestration Example in Data Engineering Fundamentals

Debug Common Workflow Orchestration Problems in Data Engineering Fundamentals

Guided Lab: Workflow Orchestration in Data Engineering Fundamentals

Data Quality and Observability: Core Concepts for Data Engineering Fundamentals

Set Up and Explore Data Quality and Observability in Data Engineering Fundamentals

Build a Practical Data Quality and Observability Example in Data Engineering Fundamentals

Debug Common Data Quality and Observability Problems in Data Engineering Fundamentals

Streaming Concepts: Core Concepts for Data Engineering Fundamentals

Set Up and Explore Streaming Concepts in Data Engineering Fundamentals

Build a Practical Streaming Concepts Example in Data Engineering Fundamentals

Debug Common Streaming Concepts Problems in Data Engineering Fundamentals

Security, Cost, and Operations: Core Concepts for Data Engineering Fundamentals

Set Up and Explore Security, Cost, and Operations in Data Engineering Fundamentals

Build a Practical Security, Cost, and Operations Example in Data Engineering Fundamentals

Debug Common Security, Cost, and Operations Problems in Data Engineering Fundamentals

Capstone Project

Build an Observable Analytical Data Pipeline

Acceptance criteria

Test your understanding

Related Programming

Check your starting point

Plan the capstone

Explore related paths