Skip to main content

Pentaho+ documentation has moved!

The new product documentation portal is here. Check it out now at docs.hitachivantara.com

 

Hitachi Vantara Lumada and Pentaho Documentation

Lineage discovery

Use the data lineage tools to help track the relationships between data resources in your data lake, which is especially helpful when you frequently merge and duplicate data. Lumada Data Catalog uses resource metadata and resource data to discover cluster resources that are related to each other. It identifies copies of the same data and merges between resources and the horizontal and vertical subsets of these resources. These relationships are the lineages of the resources.

The places where data comes into the cluster is called the origin of that data. Typically, the origin is the data source belonging to or coming from the resource. Data Catalog propagates the origin information across the lineage relationships to show the origin of the data.

Using lineage terminology

As a best practice, you may want to become familiar with the common terms used in data lineage and origins.

  • Upstream vs Downstream

    Data lineage represents the flow of data. Data comes from a source upstream and flows into a target downstream. Using the anchor resource as a vantage point, Data Catalog expresses flow in the graph from the left to right of the map.

  • Target vs Source

    A target is any resource that is pulling data from another data source. A source is any resource that feeds data out into a target. Single hop instances that are immediately upstream and downstream are known as parent and child of the target.

  • Inferred lineage

    This lineage is suggested by Data Catalog. It is inferred or derived based on the metadata and data for data resources across the platform.

  • Factual lineage

    All non-suggested lineage is considered to be factual lineage and is derived from user curations or third party tools, like Atlas or Navigator.

  • Multi-hop lineage

    Data Catalog can trace lineage up to 5 levels upstream to a source and 5 levels downstream to targets.

  • Resource-level lineage

    This type of lineage traces the relationship between the upstream to downstream resources in a data lake.

  • Field-level lineage

    For an established resource lineage, field-level lineage traces the field relationships between the source and target resources. The field-level mapping details the relationships.

  • Systems

    The system is the outermost boundary containing a resource. Every resource sits within a system. A system corresponds to a business analyst's concept of an application, library, or logical repository, such as "Sales Production" or "Corporate Warehouse". The system is key to understanding the flow of data across the firm.

    For example, if a business analyst is looking at the "Sales Team Compensation" file, then they may see a file with a sales figure upstream. If it is in the sales production system, the sales figures are probably authoritative. If they are in a training services system, they are much less authoritative.

    As lineages become longer chains, and the farther away a file is in the lineage, the user is less likely to know of the specifics of folders and such, but the system names are more likely to be informative.

    System is defined in Data Catalog as a property on a virtual folder. If the system property is not set, it defaults to the data source name. Also, a system may contain multiple resources. For more information, see Manage virtual folders for details.

Common lineage discovery tasks include:

  • Lineage tracing

    Discovering upstream data origination, or which source resources the data is coming from.

  • Impact assessment

    Discovering downstream data impact, or which target resources this data will affect.

Types of lineages

Data Catalog can import the factual lineage collected by external tools or discover inferred lineage relationships, as described in the following topics:

Imported or factual lineage

In Data Catalog, you can import the factual lineage gathered by Atlas or Navigator tools. See Lineage and origins for more details.

Inferred lineage discovery

Data Catalog's lineage discovery job discovers inferred lineage relationships among all profiled files and tables, then calculates file and table origins. Inferred lineage discovery is the backend process that is looking for the potential parent-child relationships between data resources across data sources.

Viewing lineage

Lineage is inferred in contrast with Atlas and Navigator factual lineage which is based on audit logs. Data Catalog uses a heuristic, data-centric approach to find the similar data in an enterprise. When audit logs or other historic metadata are unavailable, this method helps to discover lineage relationships. Because discovered knowledge based on heuristics is not precise, the lineage is inferred and not factual.

Inferred lineage is discovered across platform, between different data source types. The following specific parent-child relationships are targeted:

  • Copy
  • Partial copy (vertical copy)
  • Subset (horizontal and vertical copy)
  • Superset (horizontal and vertical copy)
  • Union (generally would have two or more parents, and the child resource is the union of the identified parents)
  • Partial union
  • Join

Other types of lineage, such as join/merge relationships, are likely to be discovered because Data Catalog does not require all fields to match.

Inferred lineage discovery is based on the following assumptions:

  1. Ability to match resource data for individual fields: Information is available as a result of the profiling.
  2. Ability to arrange resources based on creation date with the assumption that the parent is created before the child. This assumption results in different processing details and data source compatibility and other restrictions based on the current implementation details.
  3. For parent and child data sources, it is important to use hints to scale down computation since the enterprise is moving data from one location (data source) to another. Required hints can be target virtual folders and an optional list of parent virtual folders.
  4. Lineage is performed on regular resources and collection roots. Collection members are excluded from the lineage discovery.

Profiled data is a pre-condition to lineage discovery. Profiling must be completed for all data resources before lineage discovery.

Spark execution parameters

Inferred lineage discovery is a resource-consuming Spark application that performs across the product between different data source fingerprints. It is necessary to start with large number of executors with large amount of memory allocated for each executor.

Best practices for lineage discovery

Most Data Catalog jobs, including lineage discovery, can now be triggered from the user interface. Previously, such jobs were kicked off using the command line interface (CLI). See Managing jobs for details.

Consider the following best practices when running lineage discovery.

  • The lineage discovery job sequence operates on data in the Data Catalog HDFS metadata store. If new files are added to the cluster, run a profile job to collect profiling data so you can see information for the new files reflected in lineage relationships.
  • To optimize performance, profile all data on the cluster before running lineage discovery. During regular maintenance, run lineage discovery after a substantial number of files are added rather than running it for each incremental change.
  • Limit lineage discovery to a specific parent or child directory if your data lake is organized to allow you to isolate lineage to specific areas. Consult your administrator about modifying Spark parameters in a job if you want to discover lineages within the same folder.

    To run lineage against a set of test files, do not provide the -parentVirtualFolderList parameter. In such a case, both parent and children resources are picked from the -virtualFolder parameter listed, like in this example:

    <AGENT-HOME>$ bin/ldc lineage -virtualFolder Fin_Asia -path /data/Finance

    The Lineage command in this example triggers lineage discovery across the entire Fin_Asia folder, re-evaluating any existing suggested lineage relationships. The progress of the job is indicated by messages on the console and logged in the ldc‑jobs.log found by default in /var/log/ldc.

    NoteThis shortcut for discovering lineages within the same folder will not work for JDBC resources.
  • Lineage discovery is a memory-intensive process. As a best practice, allow for additional time to run the initial lineage discovery.

Data source specific parent-child relationship

The following table lists specific parent-child relationships.

Parent data set typeChild data set typeParent-child validation rules (ordering)
HDFSHDFSParent is older than child. Ordered by last modified date. Other configured time sensitive restrictions (see configuration.json).
HDFSHIVEFactual lineage only. Creates the lineage during HIVE schema discovery. Format discovery or UI browsing is required.
HDFSJDBCDo not use creation dates.
HIVEHDFSN/A. HDFS tables are not created from HIVE.
HIVEHIVEParent is older than child. Ordered by creation date.
HIVEDBDo not use creation dates.
DBHDFSDo not use creation dates.
DBHIVEDo not use creation dates.
DBDBDo not use creation dates. Do not allow for the same virtual folder as a parent and as a child.

Inferred lineage configuration parameters

The following table contains configuration parameters for inferred linages with default values and related resources.

ParameterNameDescriptionDefaultResource specific
batch_window_hoursMax access time windowLongest interval (in hours) between parent access time and child modification date for discovery to consider the two resources to be candidates for lineage relationships. Time checking is ignored if this value is set to 0.24HDFS
diff_same_directory_secSame directory modified time differenceShortest interval allowed between last modified dates for files in the same directory to be considered for lineage relationships. If you have a transformation process that runs on files to create similar or 'refined' copies of data in the same directory, reduce this limit to ensure that Data Catalog inventory finds lineage relationships between the original file and the modified file.30HDFS
min_lineage_field_countMin matching fieldsMinimum number of matching fields required to consider two resources as candidates for a lineage relationship.2--
min_max_cardinalityMax cardinalityMaximum cardinality value for all matched fields for lineage discovery to consider. Should be greater or equal to this value to be considered as a valid lineage.3--
overlapMin overlap valuesPortion of overlapped values to cardinality of the child field for filter (copy, partial copy) lineage or parent field for union lineage. Portion must be greater than or equal to this value to consider parent-child field as a valid candidate for a lineage relationship. This value is not recommended to change. Values lower that 0.8 may lead to a significant number of false positive matches.0.8--
filter.min_fields2inferMinimum number for mapped child fields to infer the filter lineageInfers filter lineage (copy or partial copy) if the portion of mapped child fields is greater or equal to this value.0.6--
union.min_fields2inferMinimum number for mapped child fields to infer the lineageInfers union lineage if the portion of mapped child fields is greater or equal to this value.0.8--
filter.max_alternativesMaximum number of alternative filter lineagesMaximum number of alternative filter lineages (copy and partial copy) to infer.2--
union.max_alternativesMaximum number of alternative union lineagesMaximum number of alternative union lineages to infer.2--
use_access_time_filterUse access time filterUse last-access time when checking for lineage discovery. Only applicable if the HDFS settings for last access date are enabled in the property dfs.namenode.accesstime.precision of hdfs-site.xml. Currently, this optimization is not recommended.falseHDFS
same_schemaPercent of fields with the same name to be considered as the same schemaPercent of fields with the same name to be considered as the same schema. For the same schema, lineage discovery matches the same name fields.0.8--
same_schema.non_matchingPercent of non-matching fields of same name to be excluded from the lineage considerationIf greater than this percentage of the child fields do not match the same schema, then the parent-child is not considered for lineage discovery.0.3--
same_schema.min2checkMinimum same schema fields to matchMinimum same schema fields to match. Otherwise, you cannot use same schema optimization.5--
lpupdate.batchsizeBatch size for discovery cache objectsUse this batch size to save discovery cache objects. One batch normally is saved in one HDFS MapFile in the location configured using ldc.metadata.hdfs.large_properties.uri and ldc.metadata.hdfs.large_properties.path200--
discovery.framework.right.batchsizeBatch size for left entity caching for discovery frameworkUse this batch size to retrieve left entity: first entity type defined for the discovery cross product. For tag propagation, it is a tag. For lineage discovery, it is a parent data resource.500--
discovery.framework.right.batchsizeBatch size for right entity caching for discovery frameworkUse this batch size to retrieve the right entity: second entity type defined for the discovery cross product. Currently, it is always a data resource.400--

Origins

The places where data comes into the cluster is called the origin of that data. Typically, the origin is the data source or the virtual folder belonging to or coming from the resource. Data Catalog propagates the origin information across the lineage relationships to show the origin of the data.

Origins of any resource are data sources related to this data resource directly (belongs to) or indirectly, through imported, inferred, or manually created (factual) lineages.

Origin of resource

To see origins in the single resource view, navigate to Manage, then click Tools. Select Utils from the Tools dialog box that displays. For details about entering parameters to view origins, see Examine data resource origins.