get in touch
OUR SOLUTIONS
Shorten your time to market and cut the cost of any data warehouse or data integration project using the Framework that applies design & development standards, reusable modules and software components.
Data Integration Framework enables you to have design & development standards at your disposal. Using reusable modules and software components ensures high quality and faster time-to-market for any data warehouse or data integration project by leveraging industry leading ETL/ELT tools and data storage platforms.
Using NEOS DI Framework negates non-existing development and documentation standards through the use of documentation and development standards.
Eliminates inconsistent approach to common tasks and single usage of developed components through ETL process templates and generic modules, as well as industry specific data models & reusable algorithms. Tightly integrated with different ETL Tools (OWB, ODI, Informatica, etc.).
Removes high implementation cost and risk by using proven principles and modules and based. The Framework is based on hands-on experience and tested on a number of projects.
By completely eliminating the need for different consoles, logs and monitoring tools, the framework enables easy monitoring and administration through a single console.
NEOS DI Framework is a set of different components ranging from different types as standards and architecture recommendations, predefined modules & templates to support integration with specific ETL/ELT tools, SW components supporting advanced scheduling and agents performing all kinds of running and maintenance tasks.
The reference logical architecture for Neos DI Framework is based on a number of implemented projects and best practice paradigms. This image represents a high-level overview of different areas and data flows that should be taken into consideration when designing an enterprise level data warehouse and analytics system.
Area | Purpose | Nature | Data Tracking |
---|---|---|---|
Data Staging Area (DSA) | Storing source data deltas/snapshots to be processed further in DWH Reducing load on source systems Converting to common technical platform Used also as Landing Zone (part of Data Lake) |
Source data (no transformations) Temporary storage For specific sources can be also implemented as Operational Data Store (ODS) having same structure as source data but with history/versioning included |
Standardized ETL procedures Each dataset is marked with Identifier (SSN_ID) as a reference to metadata details about source, extraction type and period and number of other information automatically loaded as part of Neos DI Framework |
Operational Data Store (ODS) | Snapshot of current data in specific data source with latest data Can be also used to store limited number of snapshots of source data Used as intermediate stage between transactional databases and DWH (EML) and/or for operational reporting purposes |
Usually data model is same/similar as one on source systems with optional extensions to store versioned data | Standardized ETL procedures Each dataset is marked with Identifier (SSN_ID) as a reference to metadata details about source, extraction type and period and number of other information automatically loaded as part of Neos DI Framework |
Enterprise Model (EML) | Industry specific common model Unified & Cleansed data definitions Logical model – Business oriented Source systems independent |
Semi-Normalized Model Storing lowest level of granularity Minimizing storage requirements Data model is based on business entities and relationships Data structure should not be affected by source system changes in most cases (except in cases additional data is introduced) |
Change Tracking (SCD2, ETL Sessions info) There are more stages ranging from storing data on lowest level of granularity to transformed unified data based on common definitions and calculations |
Target Model (TGT) | Common stage for accessing data thru reports, dashboard and ad-Hoc analysis Also, can be based on specific external consumers requirements on structure and calculations (like central bank etc) Accessed by End Users and/or external systems |
Star schema like models Specific Data Marts Shared dimensions/objects Specific export formats defined by external institutions/systems |
Loaded from EM Version Tracking Context Specific Easily Restorable and reloadable based on requirements Unified view across different datamarts |
Data Lake (DLE) | Storing large amounts of RAW data for discovery and further processing Streaming / near real-time data pipelines Staging area and Operational Data store Data archiving Complex processing involving large data sets and intensive CPU usage (simulations, data mining and similar) |
Unstructured or semi-structured datasets depending on different stages: Landing Zone: data in native format loaded as quickly and as efficiently as possible without any transformations at this stage Sandbox Zone: Partially and subject oriented interpreted, structured and cleansed data for specific domain of exploration Production Zone: Curated and described data sets with well-defined structure and purpose as a result of cleansing and transformations based on business logic. |
Each dataset is marked with Identifier (SSN_ID) as a reference to metadata details about source, extraction type and period and number of other information automatically loaded as part of Neos DI Framework |
Framework Repository (NEO) | ETL/ELT orchestration including process metadata & execution traceability/lineage Advanced Scheduling & parametrization Prerequisites & Dependencies Ensuring Data Load Traceability & Consistency |
Common metadata repository tables and views/APIs for automatic integration of data from specific ETL tool repositories Common ETL/ELT Templates & Modules ETL/ELT Tool Repository Access Layer |
All datasets are tracked by specific IDs with reference to metadata automatically loaded based on development/deployment of new objects and/or process execution results Log tables with all details on each specific change and execution More details can be found in materials related to Neos DI Framework |