Skip to main content

Pentaho+ documentation is moving!

The new product documentation portal is here. Check it out now at docs.hitachivantara.com

 

Hitachi Vantara Lumada and Pentaho Documentation

Apache Atlas integration

Parent article

Lumada Data Catalog can integrate with external data sources to share metadata using the Apache® Atlas connector. The Apache Atlas connector supports the following two job types:

  • Export: Business terms and associations in Data Catalog is exported to Apache Atlas.
  • Import: Lineage information in Apache Atlas is imported into Data Catalog.

You can creat an Apache Atlas connector by adding an external data source. For information on creating a connector, see Create an Apache Atlas connector. For information on configuring an Atlas connector, see Configure an Apache Atlas connector.

As a result, the external data source is added to the Data Sources tile in the Management page. In addition, a tab is added to the Tools page where you can import or export to Atlas. For more information on importing lineage information and exporting term associations, see the following tasks:

Create an Apache Atlas connector

Follow the steps below to create an Atlas connector for your environment.

Procedure

  1. Navigate to the Management page and locate the Data Source tile.

  2. Click External Source.

  3. Click Add External Source.

  4. Fill in the mandatory values as follows and click Create Data Source .

    Field NameValue

    External data source name

    Atlas

    External data source type

    Atlas
    URLProvide the Atlas URL
    Atlas user nameProvide the Atlas user name
    Atlass passwordProvide the Atlas password
    Atlas cluster nameProvide the Atlas cluster name

Results

The External Source is created. Additionally, the count is updated in the Data Sources tile.

Configure an Apache Atlas connector

Follow the steps below to configure the Apache Atlas connector for exporting business terms.

Procedure

  1. Navigate to the Management page and locate the Configurations tile.

  2. Click Configuration , then select Agent.

  3. Select MISC.

  4. In the Atlas connector export business term associations with status setting, enter a value of ACCEPTED, SUGGESTED, or REJECTED. You can set multiple values using a comma separation. Example: ACCEPTED, SUGGESTED.

    Configuration settingExport outcome
    ACCEPTEDOnly accepted business business terms associations are exported.
    ACCEPTED, SUGGESTEDBoth accepted and suggested business terms associations are exported.
    ACCEPTED, SUGGESTED, REJECTEDAll accepted, suggested, and rejected business terms associations are exported.

Results

After the value is entered, the configuration is updated and the business terms that have the chosen status are exported.

Importing Atlas HIVE_DB lineages to Lumada Data Catalog

This task describes how to import Atlas HIVE_DB to Lumada Data Catalog. To run an imprt job, perform the following:

Procedure

  1. Navigate to the Tools page in the left-hand navigation pane.

  2. Click External Data Source.

  3. Select Atlas as the external data source type.

  4. Enter one of the following parameters and click Submit.

    ParameterDescription

    -virtualFolder <hive_virtual_folder> -path <path>

    The import starts from the given path.

    -virtualFolder <hive_virtual_folder>

    The import starts from the root path.

  5. To complete the job, navigate to the Management page. In the Jobs tile, you can monitor the progress of the job.

Results

After the job is complete, the lineages from Atlas are imported to Data Catalog.

Exporting Lumada Data Catalog HIVE_DB level terms to Atlas

This task describes how to export Lumada Data Catalog HIVE_DB level terms to Apache Atlas. To run an export job, perform the following:

Procedure

  1. Navigate to the Tools page in the left-hand navigation pane.

  2. Click External Data Source.

  3. Select Atlas as the external data source type.

  4. Click Export.

  5. Select Atlas as the external data source.

  6. Enter one of the following parameters and click Submit.

    ParameterDescription

    -virtualFolder <hive_virtual_folder> -path <path>

    The export starts from the given path.

    -virtualFolder <hive_virtual_folder>

    The export starts from the root path.

  7. To complete the job, navigate to the Management page.

  8. In the Jobs tile, you can monitor the progress of the job.

Results

After the job is completed, business terms associated with resources from Data Catalog are exported to Atlas according to the configuration in the Atlas connector for business terms export. For example, the following table describes the possible export outcomes:
Configuration settingExport outcome
ACCEPTEDOnly accepted business business terms associations are exported.
ACCEPTED, SUGGESTEDBoth accepted and suggested business terms associations are exported.
ACCEPTED, SUGGESTED, REJECTEDAll accepted, suggested, and rejected business terms associations are exported.

If the business term exported is a built-in Data Catalog term, then you see the term as LDC_BITS_<Business_term> in Atlas.

If the business term exported is a custom Data Catalog term, then you see the term as LDC_<GLOSSORY>_<Business_term> in Atlas.

Exporting HDFS_DB level terms to Atlas

This task describes how to export Lumada Data Catalog HDFS_DB level terms to Apache Atlas. To run an export job, perform the following:

Procedure

  1. Navigate to the Tools page in the left-hand navigation pane.

  2. Click External Data Source.

  3. Select Atlas as the external data source type.

  4. Click Export.

  5. Select Atlas as the external data source.

  6. Enter one of the following parameters and click Submit.

    ParameterDescription

    -virtualFolder <hdfs_virtual_folder> -path <path>

    The export starts from the given path.

    -virtualFolder <hdfs_virtual_folder>

    The export starts from the root path.

  7. Complete the job by navigating to the Management page.

  8. In the Jobs tile, you can monitor the progress of the job.

Results

After the job is completed, business terms associated with resources from Data Catalog are exported to Atlas according to the configuration in the Atlas connector for business terms export. For example, the following table describes the possible export outcomes:
Configuration settingExport outcome
ACCEPTEDOnly accepted business business terms associations are exported.
ACCEPTED, SUGGESTEDBoth accepted and suggested business terms associations are exported.
ACCEPTED, SUGGESTED, REJECTEDAll accepted, suggested, and rejected business terms associations are exported.

If the business term exported is a built-in Data Catalog term, then you see the term as LDC_BITS_<Business_term> in Atlas.

If the business term exported is a custom Data Catalog term, then you see the term as LDC_<GLOSSORY>_<Business_term> in Atlas.