A first-hand account of the groundbreaking updates to the Data Intelligence and Quality Platforms

Collibra announced its most significant update yet at the annual Data Citizens ’24 event, introducing an abundance of new features and enhancements to their Data Intelligence Platform (DIP) and Data Quality (DQ) modules. It was amazing to experience these innovations first-hand in Orlando, Florida, and I am excited to share the key highlights with you.

What adds to the excitement is that most of the forthcoming upgrades have been thoroughly tested with several real customers in a beta programme. These features are either already available or set to roll out next month. This proactive approach in beta testing provides Collibra with a substantial competitive edge, as many vendors announce future feature releases without offering a publicly available preview. Well done, Collibra!

As anticipated, artificial intelligence (AI) was a focal point, both in the currently released features and those planned for the future. Let’s delve into what’s new at Collibra.

New UI and general features

The UI has always been a hallmark of Collibra, distinguishing it from other tools. With the new interface, Collibra has made significant strides forward. The upgrade enhances visual aspects like colours, styles, and icons, and introduces new functionality and customisation capabilities, allowing users to tailor the interface to their preferences and needs.

As we all know, there is no shortage of opinions on what a UI should contain and how it should be organised. By making the UI fully customizable, Collibra has enabled us to tailor the interface completely to our liking.

Look & Feel

The entirely redesigned UI, as depicted below, features new colours, styles, icons, and other elements, presenting a clean and modern aesthetic.

Image source: Collibra

Customization

You can easily change core elements, including logos and brand colours, via the new Customizations feature. This feature enables each client to shape their own environment according to their brand identity, offering a more cohesive user experience to their end users.

The new Customisation feature allows for significant modifications, including changing core elements like logos and brand colours, thus aligning the platform more closely with your corporate identity and enhancing the user experience. Note that styling customisations cannot be applied via CSS (a decision by Collibra that I wholeheartedly agree with, as it often leads to many unwanted side effects).

Custom Layout

Users can now personalise asset page layouts using the new layout editor. This tool supports preferred layouts and display characteristics through simple drag-and-drop operations. Each section is now equipped with an expand/collapse feature, enhancing usability and managing space more efficiently. This simple but very welcome addition will eliminate situations where a larger block of text in some asset attributes dominates the overall space on the displayed page.

Search Improvements

We can all agree that there is always room for improvement in search, can’t we? The existing search engine has sometimes proven to be too powerful, providing matching results for any resource in Collibra, such as assets, communities, domains, users, and user groups. Providing results in such a comprehensive manner sometimes offered too much information for specific user profiles using Collibra.

The enhanced search functionality now mirrors the experience found in Data Marketplace, streamlining results and focusing on search data elements. Fear not—you will still have access to all the previously existing options, but now you can easily filter and display your results in a more context-oriented way.

Usage Analytics

The key question posed by our project sponsors after the initial ramp-up period of using the Collibra platform is “Who is using the platform and in what way?”. While some level of analytics was integrated into the Collibra platform from the outset, it was more administrative in nature—not focused on the crucial details of how each user or profile is utilising the platform.

New dashboards now provide detailed insights into platform usage, addressing key queries about user engagement and behaviour patterns. This feature enhances our understanding of how different profiles are utilising the platform, which can serve as additional motivation for users to become significant contributors.

Image source: Collibra

Enterprise AI governance

AI has become an integral part of our work and daily lives, and its development and implementation across all business sectors are now essential. One of the most frequent questions we receive from clients and prospects is indeed, “How can we use AI? What can we do better with AI?”

AI Risks

The widespread integration of AI into nearly every business aspect raises substantial concerns and prompts intense discussions. Recently, numerous events have focused on the ethical, legal, risk-related, and other dimensions of AI usage.

As highlighted at Data Citizens ’24, there are many risks associated with AI deployment. A critical step in mitigating these risks involves establishing methods to systematically identify, store, and track relevant information to help minimise these risks.

Image source: Collibra

Framework

To assist us in addressing these risks, Collibra has introduced a brand-new Enterprise AI Governance framework alongside its tool/platform. This framework will enable us to easily identify for each AI initiative:

• What the business cases, users, owners, sponsors, and expected outputs are.
• What data is used and how it is utilised.
• Which models are created, where they are used, and how.
• Which policies, risks, and other aspects need to be considered.

Collibra’s AI Governance Framework is indeed a blueprint for creating successful AI products. It serves as a map to a repeatable process that drives long-term, reliable AI programs.

Image source: Collibra

Platform Enhancements

Simply put, this tool aims to enable organizations to deliver quality data to AI models and applications, ensuring that the outputs from these models and applications are reliable and can help enterprises remain compliant with emerging AI regulations.

Image source: Collibra

This is one of the key functionalities introduced, and I would be remiss not to emphasise its importance in the context of current AI initiatives. I strongly recommend that every company starts using this tool from the early stages of their AI journey.

Data Quality

The Data Quality module has also undergone a significant revamp, presenting a new interface that simplifies the navigation and management of data quality rules, job executions, and results.

The specificity of the DQ module lies in the vast amount of information related to DQ rule definition, job execution, results, and validation, among others. We all know how challenging it is to present all this information in a structured and functional manner.

Image source: Collibra

This new upgrade is a step in the right direction, introducing a simplified and guided, step-oriented interface that certainly looks promising. While I commend the new direction, I believe there is also room for improvement in future phases.

Collibra AI SQL/Rule generator

The changes in the UI will help end users be more effective while spending days investigating issues with data quality. Another welcome addition is the use of AI to make the process more effective and accessible to a wider range of user profiles. This technology enables business users, among others, to write required data quality rules using plain business language, which is automatically translated into SQL queries.

If this sounds too good to be true, rest assured. We have witnessed it working effectively, not only in English but, with the help of AI, in almost any language. Below is an example of a business user utilising the Japanese language to define the necessary rule, with the Collibra AI SQL generator successfully creating the relevant SQL query.

Image source: Collibra

DQ and DIP Integration

Data Quality and DIP Integration has been a focal point since Collibra introduced the DQ tool. We were all eager to witness a more streamlined integration of DQ with DIP, enabling us to easily view, define, and execute DQ rules directly from the DIP Assets page. Based on the updates presented in relevant sessions, there has been a significant leap forward in this area.

Users of DIP can now access all relevant details about specific asset DQ checks, the rules that are applied, the latest DQ job run information, DQ scores, and even initiate DQ job execution, all from the assets page. Once again, well done, Collibra! I look forward to seeing even more streamlined integration in the future.

Data Notebooks

I am tempted to begin another paragraph with “One of the most common questions we get from our clients…”, but isn’t that fundamentally what it’s all about? Wouldn’t we all appreciate the ability to view our real data (not just metadata), to explore, annotate, and discuss it with others looking at the same data? And to do all this within the same environment we use to manage metadata—the Data Intelligence Platform.

Image source: Collibra

Now, you can do just that directly from DIP. This innovative capability stems from Collibra’s acquisition of Husprey and marks a significant advancement in empowering data citizens to work more efficiently and collaboratively within Collibra.

Users can now query data from any source, accessed via the same EDGE installed for all other functionalities. They can write SQL queries across different data sources, create narratives, add charts, tables, and even publish such notebooks as assets.

What I would like to see—and believe should be relatively easy to implement—is the reuse of the implemented Collibra AI SQL generator to assist in creating DQ rules as an option in this module.

The overall experience

This was my first time attending the Data Citizens conference on a global scale—and I must say, I’m exceptionally satisfied not only with what was presented but also with the entire organisation.

By nature, data citizens should communicate, exchange experiences, be innovative, and discuss different approaches. All these skills were actively encouraged through various mechanisms at the conference. A few examples of how to engage participants and spark productive discussions are best illustrated by the following initiatives

Exchange your ideas and feedback for sweets.

Many visitors made new connections next to this board, trying to find all the words in the word search puzzle. Why don’t you give it a go, and try to find them all.

Conclusion

While I am generally hesitant to attend a large number of events and conferences, particularly those that require substantial time and travel, Data Citizens ’24 was undoubtedly worth the effort. Not only did I witness all these announcements, but I also met numerous interesting individuals and engaged in meaningful discussions with Collibra experts who are actively involved in the implementation and design of the newly announced features and those in the pipeline.

I was pleased to discover that Collibra uses these events not only to showcase their developments but also to engage in meaningful dialogues with all their partners and clients about what should be included in the next release and what is important to the majority. For instance, I participated in a thought-provoking brainstorming session on how we can further utilise AI. There were several intriguing ideas, and while it may seem that some may never be implemented, it is essential to remember that the ways we utilise AI today were once unimaginable. Hence, all ideas are worth considering— a topic perhaps for another blog.

Thank you, Collibra, for such a remarkable and inspiring event. Now that I am back home, we have substantial work ahead, exploring all the new features and determining how they can be employed to better assist both our existing and new clients in effectively utilising their data. See you next year!