Apache Beam and Dataflow | Build Scalable Data Pipelines

Name: Apache Beam and Dataflow | Build Scalable Data Pipelines
Availability: InStock
Rating: 4.5 (150 reviews)

Course Description

Learn how to build scalable data pipelines using Apache Beam and how data flow works in modern data engineering systems, including concepts used in Google Cloud Platform environments.

This course is intended for students who wish to master Apache Beam from scratch and also learn how to design and implement efficient data flow pipelines. In this course, students will be able to get hands-on experience building batch and streaming data pipelines and also learn how data flow works through these transformations.

Given that this course has a strong focus on practical learning, students will be able to get hands-on experience building Apache Beam pipelines using Python and Google Colab and also get an understanding of how these pipelines work using Google Cloud Platform and Dataflow.

What You Will Learn

Master the basics of Apache Beam, including pipelines, PCollections, and PTransforms

Understand the basics of dataflow, including dataflow concepts and data flow in pipelines

Develop scalable dataflow pipelines with Apache Beam

Master basic transforms, including Map, FlatMap, Filter, and Do

Master advanced transforms, including GroupByKey, CoGroupByKey, Flatten, Partition, and Combine

Master data aggregation with Max, Min, Sum, Top, Sample, etc.

Master the use of side inputs and side outputs in an Apache Beam data pipeline

Master the design of modular data pipelines with the help of composite transformations

Master the process of debugging and optimizing an Apache Beam data pipeline

Hands-On Apache Beam with Dataflow Concepts

This course is entirely practical and focuses on the development of real skills:

Learn how to build Apache Beam pipelines step by step.

Work with real data processing examples.

Understand how dataflow pipelines scale.

Get started with using Python in Google Colab.

Learn how these concepts apply to Google Cloud Platform and Dataflow environments."

Why Learn Apache Beam and Dataflow?

Apache Beam is a powerful unified programming model for building both batch and streaming data pipelines. Understanding the concepts of dataflow will enable you to create scalable systems that are applicable in modern data engineering and Google Cloud Platform.

This skill set is applicable to: