Description
Launch Your Career in Data Engineering. Master designing and implementing data solutions that use Microsoft Azure data services
This Professional Certificate is intended for data engineers and developers who want to demonstrate their expertise in designing and implementing data solutions that use Microsoft Azure data services anyone interested in preparing for the Exam DP-203: Data Engineering on Microsoft Azure. This Professional Certificate will help you develop expertise in designing and implementing data solutions that use Microsoft Azure data services. You will learn how to integrate, transform, and consolidate data from various structured and unstructured data systems into structures that are suitable for building analytics solutions that use Microsoft Azure data services. This program consists of 10 courses to help prepare you to take Exam DP-203: Data Engineering on Microsoft Azure. Each course teaches you the concepts and skills that are measured by the exam. By the end of this Professional Certificate, you will be ready to take and sign-up for the Exam DP-203: Data Engineering on Microsoft Azure.
Applied Learning Project
Learners will engage in interactive exercises throughout this program that offers opportunities to practice and implement what they are learning. They use the Microsoft Learn Sandbox. This is a free environment that allows learners to explore Microsoft Azure and get hands-on with live Microsoft Azure resources and services.
Skills measured on Microsoft Azure DP-203 Exam
Design and Implement Data Storage (40-45%)
Design and implement data storage (40–45%)
Design and develop data processing (25–30%)
Design and implement data security (10–15%)
Monitor and optimize data storage and data processing (10–15%)
The exam measures your ability to accomplish the following technical tasks: design and implement data storage; design and develop data processing; design and implement data security; and monitor and optimize data storage and data processing.
Functional groups
Design and implement data storage (40–45%)
Design a data storage structure
Design an Azure Data Lake solution
Recommend file types for storage
Recommend file types for analytical queries
Design for efficient querying
Design for data pruning
Design a folder structure that represents the levels of data transformation
Design a distribution strategy
Design a data archiving solution
Design a partition strategy
Design a partition strategy for files
Design a partition strategy for analytical workloads
Design a partition strategy for efficiency/performance
Design a partition strategy for Azure Synapse Analytics
Identify when partitioning is needed in Azure Data Lake Storage Gen2
Design the serving layer
Design star schemas
Design slowly changing dimensions
Design a dimensional hierarchy
Design a solution for temporal data
Design for incremental loading
Design analytical stores
Design metastores in Azure Synapse Analytics and Azure Databricks
Implement physical data storage structures
Implement compression
Implement partitioning Implement sharding
Implement different table geometries with Azure Synapse Analytics pools
Implement data redundancy
Implement distributions
Implement data archiving
Implement logical data structures
Build a temporal data solution
Build a slowly changing dimension
Build a logical folder structure
Build external tables
Implement file and folder structures for efficient querying and data pruning
Implement the serving layer
Deliver data in a relational star
Deliver data in Parquet files
Maintain metadata
Implement a dimensional hierarchy
Design and develop data processing (25–30%)
Ingest and transform data
Transform data by using Apache Spark
Transform data by using Transact-SQL
Transform data by using Data Factory
Transform data by using Azure Synapse Pipelines
Transform data by using Stream Analytics
Cleanse data
Split data
Shred JSON
Encode and decode data
Configure error handling for the transformation
Normalize and denormalize values
Transform data by using Scala
Perform data exploratory analysis
Design and develop a batch processing solution
Develop batch processing solutions by using Data Factory, Data Lake, Spark, Azure Synapse Pipelines, PolyBase, and Azure Databricks
Create data pipelines
Design and implement incremental data loads
Design and develop slowly changing dimensions
Handle security and compliance requirements
Scale resources
Configure the batch size
Design and create tests for data pipelines
Integrate Jupyter/Python notebooks into a data pipeline
Handle duplicate data
Handle missing data
Handle late-arriving data
Upsert data
Regress to a previous state
Design and configure exception handling
Configure batch retention
Design a batch processing solution
Debug Spark jobs by using the Spark UI
Design and develop a stream processing solution
Develop a stream processing solution by using Stream Analytics, Azure Databricks, and Azure Event Hubs
Process data by using Spark structured streaming
Monitor for performance and functional regressions
Design and create windowed aggregates
Handle schema drift
Process time series data
Process across partitions
Process within one partition
Configure checkpoints/watermarking during processing
Scale resources
Design and create tests for data pipelines
Optimize pipelines for analytical or transactional purposes
Handle interruptions
Design and configure exception handling
Upsert data
Replay archived stream data
Design a stream processing solution
Manage batches and pipelines
Trigger batches
Handle failed batch loads
Validate batch loads
Manage data pipelines in Data Factory/Synapse Pipelines
Schedule data pipelines in Data Factory/Synapse Pipelines
Implement version control for pipeline artifacts
Manage Spark jobs in a pipeline
Design and implement data security (10–15%)
Design security for data policies and standards
Design data encryption for data at rest and in transit
Design a data auditing strategy
Design a data masking strategy
Design for data privacy
Design a data retention policy
Design to purge data based on business requirements
Design Azure role-based access control (Azure RBAC) and POSIX-like Access Control List (ACL) for Data Lake Storage Gen2
Design row-level and column-level security
Implement data security
Implement data masking
Encrypt data at rest and in motion
Implement row-level and column-level security
Implement Azure RBAC
Implement POSIX-like ACLs for Data Lake Storage Gen2
Implement a data retention policy
Implement a data auditing strategy
Manage identities, keys, and secrets across different data platform technologies
Implement secure endpoints (private and public)
Implement resource tokens in Azure Databricks
Load a DataFrame with sensitive information
Write encrypted data to tables or Parquet files
Manage sensitive information
Monitor and optimize data storage and data processing (10–15%)
Monitor data storage and data processing
Implement logging used by Azure Monitor
Configure monitoring services
Measure performance of data movement
Monitor and update statistics about data across a system
Monitor data pipeline performance
Measure query performance
Monitor cluster performance
Understand custom logging options
Schedule and monitor pipeline tests
Interpret Azure Monitor metrics and logs
Interpret a Spark directed acyclic graph (DAG)
Optimize and troubleshoot data storage and data processing
Compact small files
Rewrite user-defined functions (UDFs)
Handle skew in data
Handle data spill
Tune shuffle partitions
Find shuffling in a pipeline
Optimize resource management
Tune queries by using indexers
Tune queries by using cache
Optimize pipelines for analytical or transactional purposes
Optimize pipeline for descriptive versus analytical workloads
Troubleshoot a failed spark job
Troubleshoot a failed pipeline run
