E-Books →Data Pipelines with Apache Airflow, Second Edition (MEAP V12)

Apache Airflow provides a batteries-included platform for designing, implementing, and monitoring data pipelines. Building pipelines on Airflow eliminates the need for patchwork stacks and homegrown processes, adding security and consistency to the process. Now in its second edition, Data Pipelines with Apache Airflow teaches you to harness this powerful platform to simplify and automate your data pipelines, reduce operational overhead, and seamlessly integrate all the technologies in your stack.
In Data Pipelines with Apache Airflow, Second Edition you'll learn how to
Master the core concepts of Airflow architecture and workflow design
Schedule data pipelines using the Dataset API and time tables, including complex irregular schedules
Develop custom Airflow components for your specific needs
Implement comprehensive testing strategies for your pipelines
Apply industry best practices for building and maintaining Airflow workflows
Deploy and operate Airflow in production environments
Orchestrate workflows in container-native environments
Build and deploy Machine Learning and Generative AI models using Airflow
Data Pipelines with Apache Airflow has empowered thousands of data engineers to build more successful data platforms. This new second edition has been fully revised to cover the latest features of Apache Airflow, including the Taskflow API, deferrable operators, and Large Language Model integration. Filled with real-world scenarios and examples, you'll be carefully guided from Airflow novice to expert.
about the bookData Pipelines with Apache Airflow, Second Edition teaches you how to build and maintain effective data pipelines. You'll master every aspect of directed acyclic graphs (DAGs)-the power behind Airflow-and learn to customize them for your pipeline's specific needs. Part reference and part tutorial, each technique is illustrated with engaging hands-on examples, from training machine learning models for generative AI to optimizing delivery routes. You'll explore common Airflow usage patterns, including aggregating multiple data sources and connecting to data lakes, while discovering exciting new features such as dynamic scheduling, the Taskflow API, and Kubernetes deployments.