Azure Data Factory | Data Engineering On Azure And Fabric

Data Factory on Azure and Microsoft Fabric, Azure DevOps (CI/CD), Azure Data Lake, Azure SQL and more (DP-203, DP-600)

What you'll learn

Azure Data Factory

Microsoft Fabric Data Factory

Azure Storage Accounts

Azure SQL Databases and Upsert Operations

Overview and Background of Azure Big Data Analytics Solutions

Integrating Data From Various File Formats

Authentication and Access: Service Principals, Managed Identities and Azure Key Vault

Data Pipelines, Copy Activity, Data Flows, Control Flow and Transformation Activities

Implementing a Modern Data Lake

Control Flow, Parameters and Variables

Real World Application of Azure Data Factory

Azure DevOps Source Control with Data Factory

Continuous Integration and Continuous Deployment (CI/CD)

Requirements

Microsoft Azure Subscription

Basic SQL

Description

I am pleased to present this course on one of the most in demand data engineering tools around. Azure Data Factory!As the demand for cloud-based data integration services continues to skyrocket, there is a huge demand for professionals with knowledge of services like Azure Data Factory and Microsoft Fabric. By learning these applications, users can enhance their skills and increase their job prospects in the field of data engineering and analytics.In this course you will primarily be using Azure Data Factory on Microsoft Azure and Microsoft Fabric in addition to other services such as Azure Blob Storage, Azure Data Lake Storage Gen 2 and Azure SQL Database.The course is packed with lectures, code-along videos and a dedicated course project. As an added benefit you will also have lifetime access to all of the lectures.This course will cover the following topics:Azure Storage Solutions such as Azure Blob Storage and Azure Data Lake Gen2 StorageThe basics of Azure Data Factory including the core components such as Linked Services, Datasets, Activities, Data Flows, Pipelines and Integration RuntimesIntegrating data from various file formats such as CSV, JSON and ParquetThe Copy Activity in Azure Data FactoryData Flows, Control Flow and Transformation Activities in Azure Data FactoryOrchestrating Data Integration WorkflowsHow to create Schedules and Triggers to execute your pipelinesHow to use Parameters and Variables with your Linked Services, Datasets and PipelinesHow to use Azure Data Factory with SQL DatabasesAuthentication and Access including Managed Identities, Service Principals and Azure Key VaultThe Data Factory Experience in Microsoft FabricAzure DevOps Source ControlContinuous Integration and Continuous Deployment

Overview

Section 1: Course Overview and Set Up

Lecture 1 Course Overview

Lecture 2 Azure Big Data Analytics

Lecture 3 Azure Account Set Up

Lecture 4 Azure UI Overview and Creating Your Subscription

Lecture 5 Services and Regions

Lecture 6 Naming Conventions and Creating the Course Resource Group

Lecture 7 Cost Management and Pricing

Lecture 8 Links and Resources

Section 2: Azure Storage

Lecture 9 Azure Storage Solutions Overview

Lecture 10 Classification of Data

Lecture 11 ADF Supported File Formats

Lecture 12 Customer Orders Dataset Download and Overview

Lecture 13 Blueprint for the Data Lake

Lecture 14 Storage Account Creation

Lecture 15 Container Creation and File Upload

Lecture 16 Links and Resources

Section 3: ADF - Overview and Set Up

Lecture 17 Azure Data Factory Overview

Lecture 18 ADF Resource Creation

Lecture 19 ADF User Interface and Components Overview

Lecture 20 Links and Resources

Section 4: Data Movement

Lecture 21 Copy Activity Overview

Lecture 22 Section Objective

Lecture 23 Copy Activity - Order Items Part 1 (Linked Services and Source/Sink Datasets)

Lecture 24 Copy Activity - Order Items Part 2 (Copy Activity and Pipeline)

Lecture 25 Copy Activity - Customers JSON

Lecture 26 Copy Activity - Orders Parquet

Lecture 27 Copy Activity - Stores and Products JSON

Lecture 28 Organising Data Factory Objects in Folders

Lecture 29 Chaining Activities

Lecture 30 Working with Multiple Files as a Source and Copy Behaviour

Lecture 31 Links and Resources

Section 5: Data Transformation

Lecture 32 Data Flow Overview

Lecture 33 Section Objective

Lecture 34 Data Flow Demo and User Interface

Lecture 35 Data Flow Debug (Pricing)

Lecture 36 Data Flow Debug (Row Limit)

Lecture 37 Source and Sink Partitioning

Lecture 38 Select Transformation

Lecture 39 Cast Transformation

Lecture 40 Import Projection

Lecture 41 Derived Column Transformation

Lecture 42 Raw to Cleansed Data Flow Requirement

Lecture 43 Raw to Cleansed Data Flow Walkthrough

Lecture 44 Partitioned Datasets

Lecture 45 Filter and Sort Transformations

Lecture 46 Aggregate Transformation

Lecture 47 Join Transformation

Lecture 48 Conditional Split and Union Transformations

Lecture 49 Cleansed to Structured Data Flow Requirement

Lecture 50 Cleansed to Structured Data Flow Walkthrough

Lecture 51 Structured to Analytics Data Flow Requirement

Lecture 52 Structured to Analytics Data Flow Walkthrough

Lecture 53 Links and Resources

Section 6: Scheduling and Chaining Pipelines for Execution

Lecture 54 Execute Pipeline Activity

Lecture 55 Data Flow Debug Session vs Activity Runtime

Lecture 56 Schedule Triggers

Lecture 57 Storage Events Trigger

Lecture 58 Tumbling Window Triggers

Lecture 59 Links and Resources

Section 7: Control Flow, Parameters and Variables

Lecture 60 Data Lake Preparation

Lecture 61 Delete Activity

Lecture 62 Get Metadata Activity

Lecture 63 Lookup Activity

Lecture 64 If Condition and Switch Activities

Lecture 65 Introduction to Parameters in ADF

Lecture 66 Introduction to Variables in ADF

Lecture 67 ForEach Activity

Lecture 68 Metadata Based Approach (Example)

Lecture 69 Course Project Activity: Dynamic Datasets and Folder Structure

Lecture 70 Links and Resources

Section 8: Azure SQL Database for Structured Data

Lecture 71 Section Overview

Lecture 72 Creating our Azure SQL Resource

Lecture 73 Creating our Schema and Tables

Lecture 74 Adding a Unique Identifier Column to Structured Orders Data

Lecture 75 Creating Linked Service and Datasets for SQL

Lecture 76 Important Note Regarding the Dynamic Datasets

Lecture 77 Copy Activity with SQL Sink

Lecture 78 Data Flow with SQL Sink

Lecture 79 Course Project Activity: Storing our Structured Data in the SQL Database

Lecture 80 Links and Resources

Section 9: Tumbling Window Triggers

Lecture 81 Tumbling Window Triggers Overview

Lecture 82 Ingest Data for Specific Time Intervals

Lecture 83 Tumbling Window Trigger Dependencies

Lecture 84 Links and Resources

Section 10: Authentication and Access

Lecture 85 Overview

Lecture 86 Azure Key Vault

Lecture 87 System Assigned Managed Identities

Lecture 88 User Assigned Managed Identities

Lecture 89 Service Principals

Lecture 90 Fine Grained Access Control for Storage Accounts

Lecture 91 Links and Resources

Section 11: Data Factory in Microsoft Fabric

Lecture 92 Section Overview

Lecture 93 What is Microsoft Fabric?

Lecture 94 What is OneLake?

Lecture 95 What are Fabric Capacities?

Lecture 96 Set Up a Fabric Trial with a Business Email

Lecture 97 Set Up a Fabric Trial with a Personal Email

Lecture 98 Overview of Fabric UI and Project Workspace Creation

Lecture 99 Data Pipeline and Dataflow Gen2 in Data Factory

Lecture 100 Demo 1: Copy Data Between External ADLS Containers

Lecture 101 Overview of Data Lakehouses in Microsoft Fabric

Lecture 102 Demo 2: Copy Data from Azure to Fabric

Lecture 103 Demo 3: Dataflow Gen2

Lecture 104 (Optional) Semantic Modelling and Power BI Reporting

Lecture 105 Section Debrief

Lecture 106 Links and Resources

Section 12: CI/CD: Source Control with Azure DevOps Git

Lecture 107 Section Overview

Lecture 108 Overview of CI/CD in Azure Data Factory

Lecture 109 Azure Resource Set Up

Lecture 110 Azure DevOps Overview and Set Up

Lecture 111 Azure DevOps Parallelism Request

Lecture 112 Initialising and Configuring Git Repo in ADF

Lecture 113 ADF Git Workflow Demo 1: Feature Development

Lecture 114 ADF Git Workflow Demo 2: Merge Conflicts

Lecture 115 Section Debrief

Lecture 116 Links and Resources

Section 13: CI/CD: Automated Build and Deploying to Different Environments

Lecture 117 Section Overview

Lecture 118 Reminder to Complete Parallelism Request

Lecture 119 Automated Build Pipeline

Lecture 120 Enabling Release Pipelines in Azure DevOps

Lecture 121 Release Pipeline: Deploy to Dev

Lecture 122 ARM Template Deployment Limitations: Overview

Lecture 123 ARM Template Deployment Limitations: Adding Pre and Post Deployment Tasks

Lecture 124 Test and Prod Set Up in Azure

Lecture 125 Release Pipeline: Deploy to Test

Lecture 126 Using Pipeline Variables

Lecture 127 Release Pipeline: Deploy to Production

Lecture 128 End to End Walkthrough PART 1: CI/CD with ADF and Azure Data Lake Gen2

Lecture 129 End to End Walkthrough PART 2: CI/CD with ADF and Azure Data Lake Gen2

Lecture 130 Deleting Build and Release Pipelines

Lecture 131 Links and Resources

Section 14: Continue learning with me!

Lecture 132 BONUS: Check out my other courses

Anyone interested in Azure, Microsoft Fabric or Cloud Data Engineering,Anyone interested in working with Big Data,Anyone working with cloud data