Methodology and standards
It is usual for the developer to use certain methodologies and standards as part of its project and solution implementation.
However, as part of BI Environment NEOS has further improved this approach. Here methods for design of elements used in the ELT process, such as processes and mapping and used objects (tables and definition of partitions and indices etc.) are described. This ensures correct interdependence of implementation of the ELT segment and its later daily use.
Easily reusable elements
When installing the BI Environment, NEOS also brings the library of pre-defined elements and modules that significantly accelerate mapping and processes in the ELT segment of DW systems.
These elements and modules ensure a connection between mapping and system framework (Pre-Map, Post-Map elements etc.).
Additionally, some of the elements have a more complex structure and are called pluggable modules for mapping, in which standard transformations are performed without the need for constant defining and testing. In ODI terminology such elements are called Knowledge Modules.
An often used example is closing of historical records in a slowly changeable type 2 dimension (SCD type 2), for which there is such pluggable/knowledge module typically used in all mappings for filling of dimensions where all historical changes of records must be tracked.
Prerequisites for running
The basic requirement is that the system at all times keeps data consistency as well as order of feed in line with the interdependence of data. This requirement logically results in the need to define different preconditions for running of certain processes, which can fall under different types:
- Dependence on other processes in a specific time period/segment in which the data is extracted from the source
- Dependence on receipt of files with data with the defined name
- Dependence on the result of a defined external procedure
These prerequisites are defined in a standardized manner in a unified NEOS BI Framework repository and the ETL tools are used via easily reusable elements for development of the ETL layer.
Standardized pre/post mapping procedures
The basic requirement for use of the Neos BI Environment and its upgrades for standard ELT tools functionalities is tracking of development standards of specific system components. Thus for all mappings standard pre and post mapping procedures are defined which run before and after performing of the mapping.
These procedures perform a number of tasks including:
- Recording of the performance status with detailed information
- Storage of metadata on data sets filled into specific tables
- Securing consistency of order of feeding of data sets between different areas within the data warehouse
- Logging of feeding of data sets into the staging area (feed time, extraction time, ETL procedure, feed status, process)
- Selection of the data set from the staging area in line with the order of extraction from the source (based on data from system tables)
- Marking of feed/processing status for a specific data set in system/parameter tables
- Feeding data from files of a specified name in form <PREFIX>_<DATE_AND TIME> in the order from the oldest to the newest date
Features and benefits overview
Development with NEOS BI Framework support is based on a set of standards, which ensure consistency of the ELT layer. Additionally, NEOS BI Framework provides developers with sets of pre-developed code segments (modules), which ensure integration with underlying system repository.
Main features and benefits of those segments:
- Architecture & Modelling Standards
- ETL Tool Specific Development Standards
- Best Practice Configurations
- Flexible Modelling Techniques
- Predefined pre/post ETL Module Procedures
- Best Practice Templates
- Reusable, pre-defined Pluggable/Knowledge Modules for Standardising of Complex Tasks
- Solutions Based on Best Practices
- Reusability and Improved Efficiency
- Developers Can Focus on ETL Specific Business Requirements – not on System Tasks
- Automatically Ensured Process Execution Consistency
- No Need for Additional Parameterization of ETL Process Prerequisites