Azure Data Factory +Synapse Analytics End To End Etl Project
Free Download Azure Data Factory +Synapse Analytics End To End Etl Project
Published 3/2023
MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz
Language: English | Size: 1.75 GB | Duration: 4h 16m
Complete end to end Project on OTT platform using Azure Data Factory and Azure Synapse Analytics from Scratch with Labs
Free Download What you'll learn
Create incremental pipeline based on last modified date
Calling Azure Synapse notebook from Azure Data Factory
Configure CICD from Scratch in Azure Data Factory
Build end to end project using Azure Data Engineering Services
Create a Self Hosted Integration Runtime and ingest data from On-premise Environment
How to transform data with Azure Synapse Analytics
Create and configure an Azure Data Lake
Project Oriented Learning of Azure Data Engineering services
Create an Azure Key vault service
Using Secrets of Key vault in Linked services
Create and configure an Azure SQL Database
Writing PySpark transformation logic in Synapse Notebook
Use of Triggers in Azure Data Factory
Create an incremental data load pipeline to ingest file in daily basis
Build an Automated pipeline in Azure Data Factory
Setup CICD environment from Scratch
Create incremental pipeline based on file name and pick today's file
Perform Data Analysis Using PySpark Code using Synapse Notebook
Create and configure an Azure Data Factory
Create and configure an Azure Synapse Analytics
Orchestrate all pipelines in Azure Data Factory
Loading Data into Azure SQL Database
Ingest Data from On-premise to Azure Datalake in Azure Cloud
Send an Automatic Alert email when a pipeline is failed
Transform only Today's file in Azure Synapse Analytics
Report transformed data in Power BI
Complete hands-on project using Azure Data Engineering Services
Provides insights on services that needed to clear DP-203
Requirements
Basics of Azure Data Factory
An Azure account with Subscription to perform Labs
No Synapse Analytics Experience needed, You will learn everything you needed.
Description
Are you looking to build an end-to-end ETL project using Azure data engineering services? Look no further than this comprehensive course on Azure Data Factory and Azure Synapse Analytics. This course is designed to guide you through the process of creating all the necessary services from scratch, before building an ETL project from start to finish.Throughout this course, you'll gain practical, hands-on experience with Azure Data Factory and Azure Synapse Analytics, learning how to use these powerful data engineering tools to create a highly effective ETL solution. You'll explore the many features and capabilities of these platforms, as well as their integration with other Azure services like 1. Azure SQL Database2. Azure Synapse Analytics 3. Azure Key Vault 4. Azure Data Factory for Orchestration,5. Azure Storage solutions (Azure Datalake Gen2)6. Microsoft Power BI7. Azure Logic AppsIn addition to the core ETL project, this course also includes an additional section on CICD (Continuous Integration and Continuous Deployment) on Azure Data Factory, helping you to fully automate your data engineering workflow.Whether you're a beginner or an experienced data engineer, this course is designed to help you gain a comprehensive understanding of Azure Data Factory and Azure Synapse Analytics. By the end of the course, you'll be able to confidently create and manage your own ETL projects on the Azure cloud.Enrol now and take your data engineering skills to the next level!
Overview
Section 1: Introduction
Lecture 1 Welcome to the course
Lecture 2 Main Focus and Pre-requisites
Lecture 3 Services used in this project
Lecture 4 Project Overview
Lecture 5 Project Architecture
Lecture 6 Additional section on CICD setup for Azure Data Factory
Lecture 7 Course Structure
Lecture 8 Understanding OTT dataset
Lecture 9 Resources
Section 2: Environment Setup
Lecture 10 Environment setup - Intro
Lecture 11 Creating a budget for our Project
Lecture 12 Creating a resource group
Lecture 13 Creating an Azure Data Factory
Lecture 14 Creating an Azure Datalake Storage Gen2
Lecture 15 Creating an Azure Synapse Analytics Workspace
Lecture 16 Suggestion on Saving costs for Azure SQL Database
Lecture 17 Creating an Azure SQL Database
Lecture 18 Installing Power BI Desktop
Section 3: Data Ingestion
Lecture 19 Data Ingestion - Intro
Lecture 20 Data Ingestion - Integration Runtimes
Lecture 21 Data Ingestion - What is Self Hosted Integration Runtime
Lecture 22 Overview of On-premise data source and Datalake
Lecture 23 Downloading and installing Self Hosted IR in On-premise Environment
Lecture 24 Creating and adding Secrets to Azure Key vault
Lecture 25 Creating Linked Service for Azure Key vault - Demo
Lecture 26 Creating Linked Service and Dataset for On-premise File Storage
Lecture 27 Creating Linked Service and Dataset for Azure Datalake
Lecture 28 Creating Copy Activity to copy all files from On-premise to Azure Datalake
Lecture 29 Incremental data loading using Last Modified Date of File
Lecture 30 Incremental Load based on File Name - Demo
Lecture 31 Incremental Data loading based on Filename - Practical
Section 4: Transformation
Lecture 32 Transformation - Intro
Lecture 33 Azure Synapse Analytics - Introduction
Lecture 34 Assigning Role for Synapse in Azure Datalake - Demo
Lecture 35 Assigning role and creating linked service in Azure Synapse Analytics- Practical
Lecture 36 Reading CSV files from ADLS from Synapse Notebook - Practical
Lecture 37 Stop spark session manually to save cost
Lecture 38 Identify and delete duplicate rows - Demo
Lecture 39 Identify and remove duplicate rows - Practical
Lecture 40 Identify and Remove or Replace NULL values - Demo
Lecture 41 Identify and Remove or Replace NULL values - Practical
Lecture 42 New column based on IMDB Rating - Demo
Lecture 43 New column based on IMDB Rating - Practical
Lecture 44 New column based on Runtime in Hours - Demo
Lecture 45 New column based on Runtime in Hours - Practical
Lecture 46 Practise Activity for creating a new column
Lecture 47 Solution PySpark code for Practise Activity
Lecture 48 Changing data types from String to Date Type- Demo
Lecture 49 Changing String to Date Data Type - Practical
Lecture 50 Writing transformed data to ADLS - Demo
Lecture 51 Writing transformed data to Datalake - Practical
Lecture 52 End of Writing Transformation Code
Lecture 53 Calling Synapse Notebook Activity from Azure Data Factory- Demo
Lecture 54 Calling Synapse notebook from Azure Data Factory - Practical
Lecture 55 Transformation - Conclusion
Section 5: Data Loading
Lecture 56 Data Loading - Section Intro
Lecture 57 Installing and Accessing Azure SQL Database from SSMS
Lecture 58 Loading Data to SQL Database - Demo
Lecture 59 Copying data to SQL Database - Practical
Lecture 60 Fix error in Linked service while creating for SQL Database
Lecture 61 Conclusion
Section 6: Enchancements
Lecture 62 Enhancements -Section Intro
Lecture 63 Enhancement for Copy data from On-premise to ADLS- Demo
Lecture 64 Enhancement For Copy data from On-premise - Practical
Lecture 65 Enhancement for synapse notebook - Demo
Lecture 66 Enhancement for synapse notebook to transform only todays file
Section 7: Orchestration
Lecture 67 Orchestration - Section Intro
Lecture 68 Orchestrating the pipelines- Demo
Lecture 69 Orchestrating all the pipelines and make it an automated pipeline - Practical
Lecture 70 Send an automatic alert email notification when pipeline failed in ADF - Demo
Lecture 71 Send an alert email notification when pipeline is failed - Practical
Lecture 72 Executing all pipelines for loading data into SQL
Section 8: Reporting
Lecture 73 Reporting Section- Intro
Lecture 74 Reporting Data in Power BI - Practical
Section 9: Additional Section - Configuring Continuous Integration Continuous deployment
Lecture 75 CICD - Introduction
Lecture 76 What is Continuous Integration
Lecture 77 What is Continuous Deployement
Lecture 78 CICD- Part - 1
Lecture 79 CICD - Part - 2
Lecture 80 CICD - Part - 3
Lecture 81 CICD - Part - 4
Section 10: Conclusion
Lecture 82 Conclusion for the course
Azure Data Engineers curious to do Hands-on practical Labs,Data Engineers who want real time project experience using Azure Data Engineering Services,Students aspiring to learn real time project on Azure Data Engineering
Homepage
https://www.udemy.com/course/azure-data-factory-synapse-analytics-end-to-end-etl-project/Links are Interchangeable - Single Extraction
