Pre-installation steps

The Discovery Cache stores the metadata information gathered by the Data Catalog. Data Catalog gathers metadata information about every resource it profiles in the data lake. This metadata forms the fingerprints of the data lake and are stored in the Discovery Cache in the blob storage location.

The Discovery Cache stores business term propagation and lineage discovery algorithms. Typically, this local storage is in HDFS, but in non-Hadoop deployments, a S3 bucket or other cloud storage can be used.

The Discovery Cache URI must be owned by the Data Catalog service user with full read, write, and execute privileges. Other non-service users must not be granted write permissions to this location. As a best practice, the Discovery Cache should be periodically backed up.

The size requirements are 4 KB per column.

Data Catalog uses a MongoDB database for storing audit logs and transactional data, including the sample values and sample data shown in resource views. The Data Catalog-specific data is stored in a separate collection. Since this database contains sample values and sample data, the collection should be only accessible to the Data Catalog service users.

The repository storage requirements are 2GB of MongoDB storage per 100K resources or 1M resource fields.

Data Catalog performs most of its computation as Spark jobs on the cluster. It profiles new and updated files, so after profiling existing files in a data lake, it needs resources only to manage the ingress of data.

Hitachi Vantara suggests the memory settings for the Apache YARN container as 2 GB.

The suggested Spark memory requirements are:

• Driver memory: 2 GB.
• Executor memory: Proportional to the size of the records. A size estimate is roughly 12 times the size of a single row of data being processed. For example, if your data contains rows that are 100 MB, the executor memory should be at least 1200 MB.
  NoteA row is defined as a row of data, not a single line in the file. For a JSON file that does not include line breaks, the row is the single row of data, not the entire file.
• When you run deep profiling jobs on data sets in the order of 10,000 resources, you should increase executor memory to as high as 12 GB.
• When you run business term discovery jobs on data sets of 5,000 resources, increase executor memory to 12 GB.

You can manage cluster resources that Data Catalog consumes for any given operation using YARN.

For information about tuning Spark jobs, see the Spark article https://spark.apache.org/docs/2.3.2/tuning#tuning-spark

Data Catalog stores data in the following locations:

• Blob storage

  4 KB per column

• MongoDB database

  The following example provides an estimate of the storage you would need to set up for a 250 TB data lake. It assumes an environment with the following conditions:

  • Both Hive tables and HDFS data are included in the Data Catalog.
  • All HDFS data appears in a Hive table.
  • Files and tables have an average of 30 columns, that are fields in Data Catalog.
  • Each table has a large set of backing files, such as daily files for multiple years.

The example data lake contains 2500 tables of 100 GB each. Each table has 700 files, and each table/file has 30 fields/columns.

• Total tables and files: 2500 tables with 700 files each = 1.75M resources.
• Tables: 2500 tables x 30 columns = 75,000 columns.
• Files: 2500 tables x 700 files per table x 30 columns = 52,500,000 columns.
• Total of about 53 million columns.
• HDFS storage: 53 M x 4 KB = 212 GB.

In addition, Data Catalog produces logs that are stored on the host node. A best practice is to plan for 200 MB a day and storing several weeks of logs for a total of 6 to 10 GB.

The requirements for Kubernetes include:

• Minimum of 20 GB, typically 100 GB of disk space. The node does not store data or metadata, just the software and logs.
• Minimum of 4 CPUs, running at least 2-2.5 GHz, typically 8 CPUs.
• Minimum of 6 GB of RAM, typically 16 GB.
• Bonded Gigabit Ethernet or 10 Gigabit Ethernet.
• Linux operating system. The netstat, and lsof commands are useful.
• JDK version 1.8.x.
• Installed Java Cryptography Extension (JCE) policy file available from Oracle at www.oracle.com/technetwork/java/javase/downloads/index.html.

Agents are processing engine components that can run on multiple remote clusters, and are installed on the edge nodes of a Hadoop cluster.

The requirements include:

• Minimum of 20 GB, typically 100 GB disk space. The node does not store data or metadata, just the software and logs.
• Minimum 4 CPUs, running at least 2 to 2.5 GHz, typically 16 CPUs if running Spark in Yarn client mode.
• Minimum 6 GB of RAM, typically 64 GB if running Spark in Yarn client mode.
• Bonded Gigabit Ethernet or 10 Gigabit Ethernet.
• Linux operating system. The netstat, and lsof commands are useful.
• JDK version 1.8.x.
• Installed Java Cryptography Extension (JCE) policy file available from Oracle at www.oracle.com/technetwork/java/javase/downloads/index.html.

Keycloak authentication is installed by default.