ETL Framework for Data Warehouse Environments
The non functional ETL requirements
3.91 (605 reviews)
3,943
students
6.5 hours
content
Feb 2021
last update
$54.99
regular price
What you will learn
This course provides a high level approach to implement an ETL framework in typical Data Warehouse environments. This approach can be used for a new application that needs to design and implement ETL solution which is highly reusable with data loading, error handling, audit handling, job scheduling and re-start-ability features. This framework will help reduce time and increase quality due to high re-usability and design standards.
Metadata Categories, learn the commonly used types of metadata in a real time project and how these are different from the Business and Technical viewpoints.
ETL Framework process flow, the process flow and different activities which should be taken care during the ETL framework implementation from file (source data) validations, Exception handling and Audit Control.
Data Sourcing, the different types of Data Sourcing possible in a Data Warehouse environment, different mechanisms in which the data sourcing can happen like the Scheduled events, Change Data Capture, Pub- Sub, Web services/API connectivity and the classification.
Different commonly required/used scripts for Data Sourcing, the different validations required to be performed for Data Sourcing and what functionality to be included in the scripts (shell/bat).
File Validation process, post file validation steps and file validation failure notifications.
Staging Layer, the need for staging layer, Reference Data, Audit columns for Staging and Reference tables, Data retention in the staging layer, partitions and DB standards.
Business Validation Layer, different situations possible during the data processing, concurrent workflow process, partitions in staging and business validation layer.
Data warehouse Layer, Dimension Load, Fact Load types/process, Fact partitions, Fact Summary Load and Source File Management/Archival.
Exception Handling/Error Handling, Data model for exception handling, Error Category, Error Code and different possible solutions for exception handling.
Sample Project Setup, Steps to download the project setup, executing the DDLs for metadata, project explanation and importing the code base into Informatica.
Extending the Operational Metadata’s Data Model for exception handling with additional supporting tables.
Error Handling Data Model, the framework for the data model design.
Using PMREP tables, for exception handling.
Audit, Balance and Control, the need, different technology components involved, table structure and data model, workflow example.
Configuration Management, Software Change Management, Identification, Tracking and Management of all the assets/objects of a project, One of the standard project management processes, the formal way for managing changes of the software and the process for deploying code from development to testing to production.
Related Topics
1217360
udemy ID
5/16/2017
course created date
11/22/2019
course indexed date
Bot
course submited by