Utility jobs

As a best practice, you should maintain the discovery cache by removing unused MapFile directories and merging small MapFile directories. The Data Catalog discovery cache is a repository containing large properties of entities generated during profiling and term propagation. Large properties are used in term propagation, lineage discovery, and in some instances regular expression evaluation.

The location and content of the discovery cache are controlled by configuration settings made in the Configurations user interface page. To change a configuration setting, go to Management and click View Configuration on the Configuration card. Then enter a keyword in the search box to search for a specific configuration setting.

• URI for discovery cache metadata store

  Defines the URI of the discovery cache metadata files. Its default value is s3a://ldc-discovery-cache. You should configure large properties only on the default agent.

• Relative location for a large properties metadata store

  Defines the location relative to the large properties URI for a large properties metadata store. Its default value is "/".

You should change the URI and location to fit your environment, and both should be accessible by the users running jobs.

The large properties of data resources and term propagation information in the discovery cache are stored in the resource_lp and tag_lp directories.

To view the metadata for term and large properties, enter the following command:

$ hdfs dfs -ls /user/ldcuser/.ldc_hdfs_metadata/

The output is similar to the following example:

$ hdfs dfs -ls /user/ldcuser/.ldc_hdfs_metadata/
Found 3 items
drwxr-xr-x  - ldcuser ldcuser    0 2021-09-28 05:25 /user/ldcuser/.ldc_hdfs_metadata/Built-in_Terms
drwxr-xr-x  - ldcuser ldcuser    0 2021-09-28 05:45 /user/ldcuser/.ldc_hdfs_metadata/resource_lp
drwxr-xr-x  - ldcuser ldcuser    0 2021-09-28 05:45 /user/ldcuser/.ldc_hdfs_metadata/tag_lp
$ 

Each directory contains the map files with generated unique names.

$ hdfs dfs -rm -R /user/ldcuser/.ldc_hdfs_metadata

The following configuration setting controls compression:

• Minimum number of objects in discovery cache MapFile

  If below this number, the object from MapFile will be relocated during discovery cache compression. The default is 50.

Parent Topic

• Utility jobs

Child Topics

• Compress the discovery cache

Procedure

1. Open the Data Canvas page.

2. Select the data sources that you want to investigate.

3. Click the Action menu and select Process.

4. Click Data Rationalization.

5. Add any of the following parameters and values to the Command line text box according to your requirements.

   NoteAll parameters are optional. If a value is not specified, the default value is used.

   Parameter	Description
   -firstDataSource	Specifies the first data source to use for overlap analysis. If not defined, all data sources are analyzed.
   -secondDataSource	Specifies the second data source to use for overlap analysis. If the second data source is not defined, all data sources are analyzed.
   -incremental	Specifies whether to perform incremental analysis. Set to true to perform overlap analysis for data sources only if the analysis has not been done before or requires an update. Set to false to redo the overlap analysis. The default is true.
   -reprocess	Specifies whether to reprocess data resources. Set to true to perform overlap analysis for data resources that have been processed. Set to false to perform overlap analysis only for resources that were re-profiled after the last analysis. The -incremental parameter must be set to false for this parameter to be active. The default is false.
   -fth_copy	Specifies the amount of overlap ratio when assessing a field as a copy. The formula is (1.0 - cardinality ratio < -fth_copy value). The default is 0.1.
   -fth_overlap	Specifies the precision to cut off accidental field overlaps. The file is considered an overlap if both the source and target overlap ratios are greater than this value. The default is 0.2..
   -rth_copy	Specifies the amount of precision required to detect if resources are copies. Resources are assessed as copies if the ratio of the matching fields in the compared resources is less than this value. The default is 0.1.
   -rth_overlap	Specifies the precision in overlap relationships needed to cut off accidental resource overlaps. Overlap relationships are denoted between resource1 and resource2 only if max([resource1 overlapped fields count]/[resource1 fields count], [resource2 overlapped fields count]/[resource2 fields count]) > [this value]. The default is 0.32.
   -same_semantics	Specifies whether to perform semantic analysis. Set to true to perform overlap analysis for a pair of source and target fields that have the same internally discovered semantics attributes, such as temporal, free text, numeric, or strings. To extend the scope of fields compared for overlap analysis, set this value to false. Setting this value to false may increase the accuracy of your overlap analysis, but also may cause a decrease in performance. The default is true.
   -exclude_field_list	Specifies a list of field names to exclude from overlap analysis.

6. Click Save.

Results

Next steps

Parent Topic

• Utility jobs

Procedure

1. In the left-hand navigation pane, click the Tools icon.

2. Under Encryption Tool, enter the text you want to encrypt in the Plain Text field and click Encrypt. The encrypted results display in the Encrypted result field. Plain text and encrypted texts enclosed with enc(…) are interchangeable.

   The encrypted results display in the Encrypted result field. Plain text and encrypted texts enclosed with enc(…) are interchangeable.

3. (Optional) You can copy the encrypted text by clicking the Copy Result icon.

   You can use the result in place of any plain texts. For example, use for large properties with AWS secret keys. You do not have to paste the encrypted text in the data source passwords. Passwords are automatically encrypted when they are created or updated.

4. (Optional) You can clear the plain text value by clicking Reset.

Parent Topic

• Utility jobs

You can export lineages discovered by Data Catalog and import user-defined lineages with the Lineage – Import/Export tool. You can also use this tool to delete lineages that were exported as a CSV file.

The lineage relationships, both resource level and field level, are exported in the CSV format in a pre-defined structure. Lineage import also requires the user-defined lineages to be submitted only in this pre-defined format. Lineage export and imports are based on lineage targets, not sources.

When limiting lineage export to a specific virtual folder or path, all lineages are exported to defined targets in the virtual folder or path.

Parent Topic

• Utility jobs

Child Topics

• Export-import CSV structure
• Export a lineage
• Import a lineage
• Lineage best practices
• Removing lineages

You can use the Model - Import/Export tool, a migration utility that functions like a data model mapper, to export and import entities according to the applicable environment:

• In environments with different data mappings, you can export or import the basic definitions of Data Catalog assets or the data model.
• In environments with matched data mappings, you can export or import most of the asset metadata.

The following metadata can be exported and imported:

• Data sources
• User roles
• Glossaries
• Business terms
• Term associations
• Virtual folders
• Custom properties

Parent Topic

• Utility jobs

Child Topics

• Exporting metadata
• Importing metadata

exported_termassociations.csv
exported_customproperties.csv
exported_glossaries.csv
exported_folders.csv
exported_userroles.csv
exported_sources.csv
exported_businessterms.csv

1. "Sales","Orders.Jan.Week1","Tag for Orders",VALUE,,0.0,0,0,true,true,,,""