Introduction to Data Engineering
Chapter 1: What is Data Engineering?
Defining Data Engineering
The Role of a Data Engineer
Data Engineering vs Data Science vs Data Analysis
The Data Lifecycle
Common Data Engineering Tasks
Why Data Engineering Matters for AI
Quiz for Chapter 1
Chapter 2: Foundational Concepts
Understanding Data Types
Data Sources and Collection Methods
Introduction to Databases
Data Warehouses Explained
Data Lakes Explained
Introduction to APIs for Data Retrieval
Hands-on Practical: Identifying Data Types
Quiz for Chapter 2
Chapter 3: Building Your First Data Pipeline
What is a Data Pipeline?
ETL Process Explained
ELT Process Explained
Data Extraction Techniques
Basic Data Transformation Operations
Loading Data into Storage
Simple Pipeline Orchestration Concepts
Practice: Sketching a Basic Pipeline
Quiz for Chapter 3
Chapter 4: Data Storage Fundamentals
Choosing the Right Data Storage
Working with Relational Databases (SQL Basics)
Introduction to NoSQL Databases
Understanding File Storage Systems
Object Storage Basics
Common Data Formats
Practice: Setting up a Simple Database Table
Quiz for Chapter 4
Chapter 5: Introduction to Data Processing
Batch Processing Explained
Stream Processing Explained
Processing Frameworks Overview
Understanding Compute Resources
Data Cleaning Basics
Data Validation Techniques
Practice: Simple Data Cleaning Script
Quiz for Chapter 5
Chapter 6: Essential Tools for Data Engineers
Introduction to SQL for Data Manipulation
Version Control with Git for Code
Command-Line Interface (CLI) Basics
Overview of Cloud Platforms
Introduction to Workflow Schedulers
Practice: Basic Git Commands
Quiz for Chapter 6
Chapter 7: Next Steps in Data Engineering
Areas for Further Learning
Building a Portfolio Project Idea
Contributing to Open Source
Keeping Up with New Tools
Recap of Course Concepts
Quiz for Chapter 7
Processing Frameworks Overview
Was this section helpful?
- MapReduce: Simplified Data Processing on Large Clusters, Jeffrey Dean and Sanjay Ghemawat, 2004 OSDI '04: 6th Symposium on Operating System Design and Implementation (USENIX Association) DOI: 10.1145/1070503.1070546 - The foundational paper introducing the MapReduce programming model for large-scale data processing.
- Spark: Cluster Computing with Working Sets, Matei Zaharia, Mosharaf Chowdhury, Michael Franklin, Scott Shenker, and Ion Stoica, 2010 HotCloud '10: 2nd USENIX Workshop on Hot Topics in Cloud Computing (USENIX) DOI: 10.1145/1863103.1863113 - Introduces Apache Spark, highlighting its in-memory processing capabilities for faster data analysis.
- Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017 (O'Reilly Media) - Provides a comprehensive overview of distributed systems, covering topics like scalability, fault tolerance, and various processing models (batch and stream).
- Apache Spark Documentation, The Apache Software Foundation, Current (The Apache Software Foundation) - The official documentation for Apache Spark, detailing its architecture, APIs, and use cases for various processing needs.
- Apache Flink Documentation, The Apache Software Foundation, 2025 (The Apache Software Foundation) - The official documentation for Apache Flink, providing insights into its stream processing capabilities and real-time data analysis.