Lineage
Use the Lumada Data Catalog data lineage tools to help track the relationships between data resources in your data environment, which is especially helpful when you frequently merge and duplicate data. Knowing where the data has come from can help you track down data quality problems, know whether the data can be trusted, or confirm that data from a particular system or region is included. Knowing where the data is going can help you determine who depends on it, and see how the data flows through systems and business processes.
Data Catalog uses resource data along with its metadata to discover cluster resources that are related to each other. It identifies data copies and merges resources with the horizontal and vertical subsets of these resources.
For lineage terminology definitions, see Lineage terminology. For information about lineage types, see Types of lineage.
View lineage
When you are viewing a resource in the Data Canvas, you can view the lineage information for a resource by clicking View Lineage in the Data Lineage area of the screen. This opens the Data Lineage page.
The Lineage graph visually traces relationships between resources with overlapping data.
You can use the following tools in the Data Lineage menu to help you analyze the graph and decide whether to accept or reject the traced lineages.
| Data Lineage page/Actions | |
| Label | Activity |
| Find in graph | Type the name of the resource you want to find on the current lineage graph. Suggested best matches are highlighted in the graph. |
| View | Select the type of lineages to show within the current scope of the graph, from Suggested, Accepted, and Rejected. |
| Resource view/Field view (toggle) | To show the resource lineage, click the resource icon, or to show the field lineage, click the field icon. |
| Upstream and Downstream | Click the down arrow to set the hop level in the current graph. Data Catalog can provide up to three (3) lineage hops upstream or downstream. The anchor resource is set at level 0 and is the default lineage graph for any resource for which lineage has not yet been discovered. Use the hop level to validate the authenticity of the data flowing into the anchor resource to determine whether to accept or reject the lineage. |
| Enter Focus/Leave Focus (toggle, available only in Field view) | Focused mode is selected by default when you select a field and click View Lineage. It displays the lineage graph associated with the field in focus. Click Leave Focus to view all fields and respective lineage graphs associated with the resource of the selected entity. |
| Graph controls | The graph controls include icons for zoom out, fit and center, and zoom in. Click to resize or reposition the lineage graph. |
A data lineage graph can contain the following elements:
| Graph Elements | |
| Lineage element | Description |
| Parent or source resource | Displayed per hop level. A source with a dotted boundary indicates the presence of an upstream source, while a source with solid boundary indicates the parent resource. |
| Anchor or target resource | The resource of interest for which lineage is being examined. It is indicated by a color-filled Resource node with a target icon. |
| Suggested lineage | Denoted by a dotted line between the source and target Resource nodes via an Operation node. |
| Accepted lineage | Denoted by a solid line between the source and target Resource nodes via an Operation node. |
| Rejected lineage | NoteOnly visible if the rejected lineage is set to be visible with the Rejected checkbox selected in the View element of the Data Lineage menu bar. Denoted by a red line and link icon with a diagonal line through it in the Operation node between the source and target Resource nodes. |
Lineage details
You can gain deeper insight into the lineage discovered with the details displayed on the Details pane, which change depending on whether you select the Operations, Resource, or Field node.
Operations node details
You can click the Operation node (the node with a link icon) to display the lineage information on the Details pane. The lineage actions that display vary depending on whether the lineage is accepted, rejected, or suggested, and which lineage actions are selected for viewing by the View setting in the Data Lineage menu bar.
Lineage actions
On the Data Lineage page, you can use the Details pane to accept or reject a suggested lineage or create your user-defined lineage as long as you have permission.
The Steward and Administrator roles have the permission to curate lineage.
Accept Lineage
Accept a Suggested or Rejected lineage.
Add Source
Establish your user-defined factual lineage to the operation when you specify the absolute path to a parent resource. When you add factual lineage, all suggested edges for the Operation node automatically become accepted. Factual lineage on the Operation node is then validated. Data Catalog performs path validations and actual metadata/data relation checks on your user-defined lineages on the Operation nodes, as displayed in Field Mapping details for the added resource.
Allow Discovery
Allow discovery of the resource when a lineage discovery job is run by selecting the resource and clicking Allow Discovery. Allow Discovery is a toggle with Forbid Discovery, and both are set at the resource level.
Delete Lineage
Delete an Accepted or Rejected lineage. Deleted lineages will be rediscovered on the next non-incremental Lineage Discovery job to automatically transition into Suggested lineage state.
NoteTo discover previously-deleted lineage, you need to click Allow Discovery for the resource.Forbid Discovery
Forbid discovery of a resource when a lineage discovery job is run by selecting the resource and clicking Forbid Discovery. This is useful if you want to ignore backup files, for example. Forbid Discovery is a toggle with Allow Discovery, and both are set at the resource level.
Reject Lineage
Reject a Suggested or Accepted lineage. Rejected lineages will not be rediscovered on the next non-incremental Lineage Discovery job to automatically transition into a Suggested lineage state.
View Mapping
View the field level overlap relationships between the immediate source and target resources involved in that Operation node.
Visit Resource
Go to a lineage view with this resource as the target resource.
Rejecting or deleting edges on the Operations node
Lumada Data Catalog does not support lineage actions for independent edges. Any action on an edge will be performed on the Operation node. Exercise caution when adding factual sources on an Operation node. These sources cannot be independently rejected or deleted without affecting the other resources associated with the operation.
To remove a factual source from an operation node:
- Reject the operation.
- Delete the operation.
- Re-run lineage discovery to recover any suggested lineages associated with the Operation node that you deleted in step# 2.
Additional lineage information
You can find additional information about the lineage on the Details pane. The panels visible depend upon the element selected in the lineage graph.
Actions
Depending on the element selected, you can:
- Accept Lineage
- Add Source
- Allow Discovery
- Delete Lineage
- Forbid Discovery
- Reject Lineage
- View Mapping
- Visit Resource
Allow Discovery
Allow discovery of the resource when a lineage discovery job is run.
Code
Populated by Data Catalog with a default expression used to identify the relationship for the selected Operation node.
Comment
Notes related to the lineage.
Description
Description of the lineage node.
Expression
Placeholder for a future Data Catalog use.
Forbid Discovery
Forbid discovery of certain resources when a lineage discovery job is run.
Glossaries
Glossaries, if any, related to the lineage element.
History
Displays timestamps for the chronological history of the time of creation and time of last modification for the lineage operation.
Resource
Name of the resource.
Resource Terms
Terms, if any, related to the resource. You can click Add Term to assign an existing term to the resource.
Resource node details
On the Data Lineage page, click the Resource node to display the resource-related lineage actions and information on the Details pane.
Resource Type & Path
The Code section identifies the type of resource (file/collection/table and so on) with the absolute path.
Actions
If your user role allows, you can use the following actions depending on whether the resource is a target (anchor) resource, or a source resource:
Add Source
For a target (anchor) resource, you can add and define factual lineage by specifying the absolute path to a parent resource. For a source resource, you can visit the resource.
Resource Terms
Lists any resource business terms associated with that resource. Users with permission can click Add Term to add a term.
Glossaries
Lists the glossaries of any field business terms associated with or suggested on the fields of that resource.
Description
The plain text description of the resource that is obtained from the Summary tab.
History
Displays the time created and time last modified timestamps for the resource.
Field node details
On the Field node, you can select the target (anchor) resource from the drop-down menu to display the lineage for the field between the source and target resources. The details for the Field node provide the field-related information. If a node is selected, its status is shown (Suggested, Accepted, or Rejected).
Actions
You can take the following recommended actions depending on whether the resource is a target (anchor) field, resource, or a source resource:
Add Lineage
For a target (anchor) field, you can manually add lineage relationships with other fields. Select the field, right-click and select Add Lineage, then select the field from the navigation tree or pop-up (multiple field selection is also possible). It creates an accepted lineage graph since it is manually added.
Forbid Discovery or Allow Discovery
For a target (anchor) field or resource, you can select Forbid Discovery or Allow Discovery (toggle). Forbid Discovery forbids the resource from being found from the next lineage job run. Allow Discovery allows that resource to be found when a user runs the lineage job again.
Add Source
For a target (anchor) resource, you can select Add Source and add the source to it.
Visit Resource
For a source resource, you can select Visit Resource, and it takes you to the resource view.
NoteYou can also perform various actions or curations like Accept Lineage, Reject Lineage, and Delete Lineage on the field level, similar to resource level lineage.Description
Refers to the flattened text description associated with a field as set via Rest API or Hive/JDBC comments.
Glossaries
Lists the glossaries of Field Terms associated with the resource field.
History
Displays the chronological history of the time of creation and the time of the last modified timestamps for the resource.
Importing lineages
Lumada Data Catalog provides a framework to allow integration to other applications from email servers to data cleansing and visualization tools. The integration framework lets you define an action menu option at the resource level that will initiate a client-side or server-side operation.
With the help of this framework, you can import the lineages from third-party tools.
Integrating the Atlas third-party tool
In Data Catalog, you can integrate the Apache Atlas third-party tool to import lineages. To import lineages, click Tools on the left navigation menu and click Lineage – Import/Export. Upload a file with lineage information that you want to import. From the drop-down menu, select Import Operations from Atlas.
After you import the operations, you can view the imported lineages by selecting View Lineage from a resource selected in the Data Canvas. The Description displays Atlas Lineage Import. All the imported lineages are in the Accepted status.