Apache Atlas integration
Lumada Data Catalog can integrate with external data sources to share metadata using the Apache® Atlas™ connector. The Atlas connector supports the following two job types:
- Export: You can export business terms and associations from Data Catalog to Atlas.
- Import: You can import lineage information in Atlas into Data Catalog.
You can create an Atlas connector by adding an external data source. For information on creating a connector, see Create an Atlas connector. For information on configuring an Atlas connector, see Configure an Atlas connector.
As a result, the external data source is added to the Data Sources tile in the Management page. In addition, a tab is added to the Tools page where you can import or export to Atlas. For more information on importing lineage information and exporting term associations, see the following tasks:
- Import Atlas HIVE_DB lineages to Data Catalog
- Export Data Catalog HIVE_DB level terms to Atlas
- Export HDFS_DB level terms to Atlas
Create an Atlas connector
Procedure
Click Management in the left navigation menu and on the Data Sources card, click Add New and select Add External Data Source.
Fill in the mandatory values as follows and click Create Data Source .
Field Name Value External data source name
Provide the name of the data source External data source type
Provide the Atlas URL, including the host and port for the Atlas service Atlas user name Provide the Atlas user name Atlas password Provide the Atlas password Atlas cluster name Provide the Atlas cluster name Test the connection.
Results
Configure an Atlas connector
Procedure
Click Management in the left navigation menu.
The Manage Your Environment page opens.On the Configuration card, click View Configuration.
Under local-agent Categories, click View Details (down arrow at the end of the row) for MISC.
Expand the Atlas connector export business term associations with status setting by clicking the down arrow.
Enter a value of ACCEPTED, SUGGESTED, or REJECTED. You can set multiple values by separating values with a comma. Example: ACCEPTED, SUGGESTED.
Configuration setting Export outcome ACCEPTED Only accepted business terms associations are exported. ACCEPTED, SUGGESTED Both accepted and suggested business terms associations are exported. ACCEPTED, SUGGESTED, REJECTED All accepted, suggested, and rejected business terms associations are exported. Click Save Changes.
Results
Import Atlas HIVE_DB lineages to Data Catalog
Procedure
Click Tools in the left navigation menu.
Click the External Data Source tab.
Atlas is the default external data source type.If it is not selected, click Import.
Select the external data source from the drop-down list.
Enter one of the following parameters and click Submit.
You can monitor the progress of the job on the Job Activity page.Parameter Description -virtualFolder <hive_virtual_folder> -path <path>The import starts from the given path. -virtualFolder <hive_virtual_folder>The import starts from the root path.
Results
Export Data Catalog HIVE_DB level terms to Atlas
Procedure
Click Tools in the left navigation menu.
Click External Data Source.
Atlas is the default external data source type.Click Export.
Select the external data source from the drop-down list.
Enter one of the following parameters and click Submit.
You can monitor the progress of the job on the Job Activity page.Parameter Description -virtualFolder <hive_virtual_folder> -path <path>The export starts from the given path. -virtualFolder <hive_virtual_folder>The export starts from the root path.
Results
| Configuration setting | Export outcome |
| ACCEPTED | Only accepted business terms associations are exported. |
| ACCEPTED, SUGGESTED | Both accepted and suggested business terms associations are exported. |
| ACCEPTED, SUGGESTED, REJECTED | All accepted, suggested, and rejected business terms associations are exported. |
If the business term exported is a built-in Data Catalog term, then you see the term as LDC_BITS_<Business_term> in Atlas.
If the business term exported is a custom Data Catalog term, then you see the term as LDC_<GLOSSORY>_<Business_term> in Atlas.
Export HDFS_DB level terms to Atlas
Procedure
Click Tools in the left navigation menu.
Click the External Data Source tab.
Atlas is the default external data source type.Click Export.
Select the external data source from the drop-down list.
Enter one of the following parameters and click Submit.
You can monitor the progress of the job on the Job Activity page.Parameter Description -virtualFolder <hdfs_virtual_folder> -path <path>The export starts from the given path. -virtualFolder <hdfs_virtual_folder>The export starts from the root path.
Results
| Configuration setting | Export outcome |
| ACCEPTED | Only accepted business terms associations are exported. |
| ACCEPTED, SUGGESTED | Both accepted and suggested business terms associations are exported. |
| ACCEPTED, SUGGESTED, REJECTED | All accepted, suggested, and rejected business terms associations are exported. |
If the business term exported is a built-in Data Catalog term, then you see the term as LDC_BITS_<Business_term> in Atlas.
If the business term exported is a custom Data Catalog term, then you see the term as LDC_<GLOSSORY>_<Business_term> in Atlas.