Data Catalog overview

Lumada Data Catalog provides a rich interface for data self-service by leveraging multi-faceted metadata search using tags, detailed properties, bookmarks, and Hive tables to help you find the best instances of the integrated data you are looking for. These services include:

• Role-based access control

  Administrators can assign a role the permission to view the metadata for files and fields without assigning the permission to view the data. The ability to view just the metadata is useful because many times data analysts do not know they need access to a file until they have access to it.

  When administrators create roles, they designate the resource read access level for every role using the following settings:

  • Metadata Access

    This parameter is used to determine whether to override the system permissions such that the selected role members are allowed read-only access permission to the resource's metadata.

  • Data Access

    This parameter is used to determine the read access to a resource.

• Multi-faceted, customizable metadata search

  During profiling and discovery, Data Catalog infers facets from both data and metadata properties. You can use facets to refine search results and drill down to the desired file, table, or field without having to write code or ask subject matter experts. You also can create facets from cluster data.

  You can use pre-filters for a certain set of values to customize facets for a specific role, so you can search within a narrower scope for the values or facets relevant to their function.

• Tags

  Data Catalog allows you to discover the content of files and tag fields automatically. It also provides the ability to import an ontology vocabulary so users can leverage pre-defined terms, as well as by crowd-sourcing the creation of tags. Data Catalog learns from the way users tag the data, and propagates tags to similar fields automatically.

• Data quality metrics and statistics

  Data Catalog's profiling processes produce and present detailed data quality metrics and statistics that help you decide if the data is useful, valid, and complete, without having to write code to graph each field in the file.

• Bookmarks and audit history

  As you search for applicable files and tables to use, Data Catalog helps you keep track of these resources with bookmarks. You can use bookmarks to create a shortcut for browsing directly to the bookmarked resource. Additionally, bookmarks also register your interest in that resource, so you are notified when changes to the resource, such as schema changes or tag changes, are detected.

  In addition to notifying users, Data Catalog keeps an audit history of all user actions and resource changes on the cluster.

• Hive table creation

  When you find a file you are interested in, you can easily create a Hive table for that file. This ability is useful for non-technical users to find the files that contain the data that they need, create a Hive table and use a popular business intelligence (BI) tool to visualize the data without any added efforts, making Hadoop accessible and useful to a much wider audience.

• User roles

  User roles help you manage tags in multiple domains and provide control of the users who are making the changes in each domain. You can open up one domain to ad hoc tagging while keeping another domain under strict curation.

• Job management

  You can delegate job management based on user roles, which allows the owner of a data node or resource, such as a data steward or data analyst, to make administrative decisions for and take ownership of that node or resource.

Behind Data Catalog’s self-service user interface is an engine that profiles the data repository and enriches it by propagating tags created by users. Data Catalog identifies the formats of the resources and profiles their contents, creating an inventory of data assets in the data warehouse automatically and securely.

• Format discovery

  One of the benefits of a data warehouse is that it is a central repository of integrated data from different data source types with different types of files and tables, without a requirement for the data to fit a predefined resource schema. However, this flexibility can also make it difficult to understand the structure of the resource and how key fields may be related within and across resources.

  Data Catalog crawls a data cluster and identifies the formats of a wide variety of resources, including images, PDFs, logs, data files, and tables. All the recognized resources are added to the inventory for tagging, searching, and lineage.

• Profiling

  Most of the data-curating process entails writing code to profile and graph data. Data Catalog automates this process, improving the productivity of data engineers and data scientists.

  Data profiling is the process in which Data Catalog examines file data and gathers statistics about the data. It profiles data in the cluster, and uses its algorithms to compute detailed properties, including field-level data quality metrics, and data statistics. The resulting inventory includes rich metadata for delimited files, JSON, ORC, AVRO, and Parquet and from files compressed with supported compression algorithms such as gzip, LZO, and Snappy.

• Lineage discovery

  File lineage is the origin of a file or the systems it came from. Knowing the lineage across files is important for audit purposes, to determine if a file is trustworthy, and to do impact analysis to determine how schema changes would affect downstream files. Data Catalog performs file lineage discovery automatically, so you can view and drill into the lineage information as you work with the data.

• Sensitive data discovery

  Sensitive data residing in the data cluster presents a sizable liability if it is not protected and managed. Data Catalog’s algorithms identify sensitive data throughout the data clusters as a part of profiling with minimal additional overhead. Identification is the first step, and often the hardest step, in the process of protecting sensitive data. You cannot protect sensitive data unless you know where it resides. Data Catalog identifies sensitive data and facilitates the next step of protecting it through masking, encryption, or quarantine.

• Tag propagation

  Tag propagation is central to the Data Catalog's ability to crowd-source an ontology. Tag propagation is the process in which Data Catalog learns from the way users tag the data, recognizing similar content across resources and automatically assigning tags to similar fields.

  Tag propagation allows all users to benefit from the insight and knowledge that other users encapsulate in the tags they created. The end result is a rich understanding of the data that was achieved by the community with little effort.

You can discover data quality metrics automatically using large scale profiling, such as discovering the number of nulls in a data column or cardinality. For example, you can assess the number of values that should be in a field versus the actual numbers that have been profiled.

You can search these statistical demographics in the user interface, with the Lumada Data Catalog RESTful API, or extract them to a reporting tool. You can also use the API to integrate Data Catalog with a data quality tool.

Data Catalog writes profiling process notifications to the log files.

Reporting in Data Catalog usually is done using third-party BI tools. Data Catalog provides a RESTful API for the reporting tools to consume, so nearly any integration can be performed in the most standard format.

Data Catalog complements data visualization, data discovery, and data wrangling tools by streamlining the collection and initial data quality checks of the data repository. Then it makes the data repository available to those tools for further processing.

With Data Catalog, you can tag reports in the same way you tag any raw data, whether the data is for a report, an ERP, or a CRM dataset.

For reporting, you can easily connect to your BI tools from Data Catalog using the REST API. For example, you can use Data Catalog's REST API with Tableau to effectively query the Data Catalog.