In ETL world where data is in constant flux two ingredients are necessary for success. Moving data correctly, in context, and scheduling it to move at the right time. ETL tools implement move and transformation of data. Neos DI Framework makes sure data is moved only when it is supposed to and when its dependencies are ready. In the next few paragraphs we describe ETL evolution path and explain why was Neos Framework crafted in the first place.
When businesses first start, data is low in volume, sales report deadlines comfortable and knowledge of data manipulation scarce. Copying, transforming and generating reports in this stage is an easy task usually done manually with scripts.
After businesses expand and its data multiplies, script maintenance becomes bottleneck and more people are hired. More people, more diversity. Diversity is good for civilization, but not for data manipulation. Same things done differently, reinventing the wheel, losing control of a bigger picture. When this happens, usually local hero is called – graphical ETL tool.
Graphical ETL tool makes things easier on many levels. It standardizes solutions for common data manipulation problems. It encourages focus on what should be done with data, leaving how it’s done to ETL tool. It enforces reuse and helps in understanding the connections between data objects.
Main characteristic of ETL tool is allowing data manipulation in three stages: extracting, transforming and loading. Data is extracted from the source system(s), transformed if necessary and loaded into target system for further analysis or additional transformation. Data starts to flow between systems and these flows need to be synchronized. First data-pipelines are manually created and monitored. Productivity is again on the rise – at least until ETL beast awakes.
Getting things done quicker doesn’t mean less work in the future. Just the opposite – it means more demanding users. Those users create new use-cases for data and that results in reorganization of ETL. And as more and more data-pipelines are created, it begins to complicate scheduling of these data flows. Questions like “Can I load this data now or after 5pm?” or “Why I don’t have yesterday sales available?” are common and usually responded with “Don’t know – there are too many dependencies” and “Our system spend too much time in idle”. Also, there is a challenge of technology improvement – better ETL tools are created and this can lead to change of ETL tool. Change can be a new version of the same tool or completely different tool.
All these challenges – new data sources and use-cases, complicated scheduling, data dependency and ETL tool evolution – cost time, money and end-user experience. To overcome these challenges and allow faster time-to-market implementations, Neos DI Framework was introduced in the ETL realm.
Neos Framework was first created in need of optimization of ETL data flows. Instead of running ETL loads on time-based schedule, Neos Framework introduced automatic, dependency scheduling – when all prerequisites for some ETL flow are finished successfully, that flow is automatically started. Building on that, Neos Framework introduced best practices and ETL methodology standardization and added new features to add more value to the ETL projects.
After ten years of continuous improvement, Neos DI Framework is battle-tested in production and ready to serve you in your next ETL project. It has become large, but modular and understandable platform which is capable of supporting large development teams. They all adhere to the same standards making development of complex ETL/ELT solutions as streamlined as possible. It supports different versions of Oracle and non-Oracle ETL tools. It even allows to use those tools in parallel, enabling organic evolution of the Enterprise DWH systems. Adopt new tool for new modules and projects, and gradually decommission older process as they become obsolete.
Neos DI Framework consists of quite some modules, so please explore other texts to gain deeper insight.
And of course – feel free to contact us for any details on how NEOS DI Framework can help with your specific situation.
FinOps – Cultural Transformation
Rethinking your procurement analytics – Start spending smarter
Our management doesn’t really understand or support DevOps. What can I do
How does NEOS DI Framework support Big Data technologies – Hadoop example