Installing Lumada Data Catalog on Amazon EMR

Procedure

1. Execute the following command::

   ./ldc-app-server-*.run The following text displays in the Terminal window:

   Verifying archive integrity...  100%   MD5 checksums are OK. All good.
   Uncompressing Lumada Data Catalog App Server Installer  100%
   
   This program installs Data Catalog Application Server.
   
   Press ^C at any time to quit.
   
   #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                  LUMADA DATA CATALOG APPLICATION SERVER INSTALLER
   #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   Enter the name of the Data Catalog service user [ldcuser]: wlddev
   Enter install location [/opt/ldc] :
   Enter log location [/var/log/ldc] :
   #~~~~~~~~~~~~~~~~~~~~~~~
      SELECTION SUMMARY
   #~~~~~~~~~~~~~~~~~~~~~~~
        Data Catalog service user : wlddev
                   Install location : /opt/ldc (will be created)
                       Log location : /var/log/ldc (will be created)
   Proceed? [Y/n]:
   [sudo] password for wlddev:
   Created directory /opt/ldc
   Created directory /opt/ldc/app-server
   Created directory /var/log/ldc
   Copying files ... done.
   
   Installed app-server to /opt/ldc/app-server
   Starting services ............ done.

2. Open a browser window, then navigate to the setup link at http://<ldc node>:8082/setup.

   The browser opens the Welcome to Lumada Data Catalog page.

3. Click Let's Get Started. The setup wizard opens.

   The End User License Agreement page appears.

4. Read the license terms and conditions and select the check box to accept the license agreement. Click I agree.

   The Connect with Solr page appears.

5. On the Connect with Solr page, enter the following fields and settings to set up the Data Catalog Solr collection repository:

   1. In Solr Client Mode, choose the client mode corresponding to your Solr installation.

      For EMR, a best practice is to use Cloud mode.

      Use the same values that you configured in Installing Solr and create a collection.

   2. In the Solr Server Url field, enter the URL of Solr server.

      Review the Solr connection URL to ensure it matches the location of the Solr server. The default is the same node where Data Catalog is installed.

   3. In the Solr Zookeeper Ensemble field, enter the ZooKeeper ensemble.

   4. In the Lumada Data Catalog Collection Name field, enter the Solr collection name as defined in the previous Solr steps.

   5. In Solr Authentication Mode, select an authentication mode: None, Basic, or Kerberos.

   6. Click Test Connection.

      The Connection Successful message appears.

      If the test fails, make sure that Solr is running and that the Data Catalog service user has access to the collection.

   7. Click Next step.

   The Connect with Postgres page appears.

6. Enter the following fields and settings to set up the Postgres database, which is used for audit logs and discussions:

   1. In the Postgres Driver Class field, enter the class used to communicate with the Postgres database.

   2. In the Url field, enter the location of the Postgres database installation.

   3. In the Postgres User field, enter the username used to access Postgres.

   4. In the Postgres Password field, enter the password for the above username used to access Postgres.

   5. Click Test Connection.

      The Postgres Connection Successful! message appears.

      If the test fails, make sure that Solr is running and that the Data Catalog service user has access to the collection.

   6. Click Next step.

   The Large properties storage page appears.

7. Enter the following fields for the location in your cluster, which are typically HDFS. These fields are used to store metadata information required for running jobs and identifying tags.

   1. In the Large Metadata Storage Uri field, enter the fully-qualified URI location to store intermediate metadata for Lumada Data Catalog processing jobs.

      This URI will be automatically detected by the installer. However, if incorrect, enter the URI of the HDFS name node. If HA is enabled, this value is the HA URI of the HDFS service.

      Set the storage location URI if different than the local HDFS. For example, this location could be an S3 bucket.

   2. In the Parent Path field, enter the path of the parent where you want to store the metadata, which is typically the home directory of the Data Catalog service user.

      This path must be write-accessible by the Data Catalog service user. Subsequently, when running jobs, a directory named .ldc_hdfs_metadata is created under this path.

   3. Click Test Connection.

      The Connection Successful message appears.

      If you have not already configured the Hadoop proxy settings of the Data Catalog service user, the Test Connection might fail. Follow these steps and make sure the client configuration is applied to the entire cluster and that the cluster has been restarted.

   4. Click Next step.

   The Repository Bootstrap page appears.

   Data Catalog begins the repository and roles bootstrap process. Make sure that the Solr schema has been created, the Postgres schema has been created and roles, job sequences and built-in tags are bootstrapped successfully. You should only have to create the Solr collection creation once.

8. When all processes are complete and display a checkmark, click Next step to continue.

   The Authentication method page appears.

9. Select and configure your user authentication scheme.

   Select the authentication method that allows Data Catalog to validate users who log in to the web application.

   For EMR instances, a best practice is an LDAP server. The following steps are for using LDAP authentication to validate the users logging into Data Catalog.

   1. In Authentication type, select the authentication type. For LDAP enter the following fields.

   2. In LDAP Auth Mode, select an authentication mode: bind-only, bind-search, or search-bind.

      See LDAP search modes for more information.

   3. In the LDAP Url field, enter the URL for the authentication type.

      The default entry is a free, third-party LDAP provider. The URL begins with ldap://. If you are using a secure connection to the server, the URL begins with ldaps://. The standard LDAP server port is 389 and 636 for SSL.

   4. In the Auth Identity Pattern field, enter the identity pattern for the authentication.

      The pattern must contain the username that will replace the actual user ID.

      This string must include the phrase "uid={USERNAME}" and can include other LDAP configuration parameters such as to specify users and groups. You can add a search root to the URL to restrict user identity searches to only a part of the LDAP directory.

      For example, "uid={USERNAME},dc=subsidiary,dc=com"

   5. In the Lumada Data Catalog Administrator field, enter a user as the administrator who will manage the Data Catalog.

      It is recommended to enter ldcuser. However, you can enter a different name here. This user is granted the Data Catalog administrator role, which is configured for access to all data sources and tag domains.

      Use this login to add additional users and continue configuration tasks.

   6. In Test Authentication, enter the user credentials of the administrator in the Username and Password fields.

   7. Click Test Login.

      The Login Successful message appears.

   8. Click Next step.

10. On the last step of the setup wizard, copy the Metadata Server installation command from the Metadata REST server details page, but do not execute it yet. Then click Next step.

    You need this information when installing the LDC Metadata Server. The LDC Application Server installation automatically creates the token for the LDC Metadata Server, which is used for initializing and registering the Metadata Server with the Application Server.

    

    NoteThe same Metadata Server token is available in the user interface after restarting the Application Server. Select Install Metadata Rest Server, under Manage Tokens metadata-rest-server and select Install Metadata Rest Server.
    The Restart page appears.

11. Click Restart to apply the changes.

    You may have to restart the Data Catalog services through the command line to make sure the changes are applied successfully. After the changes are applied, Data Catalog is ready.The Welcome page appears.

Next steps

Parent Topic

• Downloading the Data Catalog packages

Procedure

1. Restart the Application Server.

2. Execute the following command on the node where you want to install the Metadata Server.

   ./ldc-metadata-server-*.run -- --init --endpoint proton:8082 \
   --client-id metadata-rest-server \
   --token 4236cea0-93ad-416d-9b38-919392ac6059 \ 
   --public-host proton \
   --port 4242
   

   Refer to the following list for a description of each argument:

   • --init

     Initialize: synchronize the repository configuration from the LDC Application Server.

   • --endpoint

     The URL of the LDC Application Server you want to connect to.

   • --token

     Authentication token.

   • --public-host

     Public host of the LDC Metadata Server to be reported to the LDC Agent when it subsequently registers. "Public" does not necessarily mean the internet facing public hostname/IP. It only means the hostname/IP that is routable from all the LDC Agent. If all the LDC Agents are part of a private subnet, then enter the private hostname/IP of the LDC Metadata Server host.

   • --port

     Port on which to run.

   The LDC Metadata Server installer is verified and extracted.

   The following text displays in the Terminal window:

   Verifying archive integrity...  100%   All good.
   Uncompressing Lumada Data Catalog Metadata Server Installer  100%
   
   
   This program installs Lumada Data Catalog Metadata Server.
   
   Press ^C at any time to quit.
   
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                  LUMADA DATA CATALOG METADATA SERVER INSTALLER
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       1. Express Install          (Requires superuser access)
       2. Upgrade
       3. Exit
   
   Enter your choice [1-3]: 1
   Enter the name of the Lumada Data Catalog service user [ldcuser]:
   Enter install location [/opt/ldc] :
   Enter log location [/var/log/ldc] :
   Enter the Solr server version [8.4.1]:
   Is Kerberos enabled? [y/N]: y
   Full path to Lumada Data Catalog service user keytab : /home/ldcuser/ldcuser.keytab
   Lumada Data Catalog service user's fully qualified principal : ldcuser@HITACHIVANTARA.COM
   ~~~~~~~~~~~~~~~~~~~~~~~
      SELECTION SUMMARY
   ~~~~~~~~~~~~~~~~~~~~~~~
   Lumada Data Catalog service user : ldcuser
                   Install location : /opt/ldc
                       Log location : /var/log/ldc
                   Kerberos enabled : true
               Kerberos keytab path : /home/ldcuser/ldcuser.keytab
                 Kerberos principal : ldcuser@HITACHIVANTARA.COM
                Solr server version : 7.5.0
   Proceed? [Y/n]: y
   Removed existing directory /opt/ldc/metadata-server
   Directory /opt/ldc exists.
   Created directory /opt/ldc/metadata-server
   Directory /var/log/ldc exists.
   Copying files ... done.
   
   Installed metadata-server to /opt/ldc/metadata-server
   Generating certificate ...
   SLF4J: Class path contains multiple SLF4J providers.
   SLF4J: Found provider [org.slf4j.simple.SimpleServiceProvider@38af3868]
   SLF4J: Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@77459877]
   SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
   SLF4J: Actual provider is of type [org.slf4j.simple.SimpleServiceProvider@38af3868]
   [main] INFO com.hitachivantara.cli.WldSSLCertificateGenerator - Absolute path for keystore is : /opt/ldc/metadata-server/conf/keystore
   [main] INFO com.hitachivantara.utils.WldSSLUtility - SSL certificate successfully generated. Storing certificate in keystore
   [main] INFO com.hitachivantara.utils.WldSSLUtility - SSL certificate successfully stored in keystore
           Certificate fingerprint (SHA-256): 28baed0ff68461d6079e8faccb7132d835abb1f66589b2c6d11dcbd313c69f12
   Executing command: "/opt/ldc/metadata-server/bin/metadata-server" init --endpoint http://ec2-xx-xxx-xx-xx.aa-aaaa-1.compute.amazonaws.com:8082 --client-id metadata-rest-server --token d60766a6-7c5c-49e5-b86e-dd759fd640eb --public-host ec2-xx-xxx-xx-xx.aa-aaaa-1.compute.amazonaws.com --port 4242 --no-exec false
   Initializing application
   done.
   removed ‘/tmp/tmp.UqOj0eBmhY’

Next steps

Parent Topic

• Downloading the Data Catalog packages

Procedure

1. In Lumada Data Catalog, navigate to Manage Agents.

   The Agents page opens.

2. Click Create Agent.

   The Create Agent dialog box opens.

3. Enter a name and description for the Agent in the Name and Description fields, then click Add.

   

   The Register Agent dialog box opens.

4. Copy the command generated in the Register Agent dialog box.

   

5. Run the copied command to install the LDC Agent as follows:

   ./ldc-agent-*.run -- --register --endpoint http://ec2-xx-xxx-xx-xx.aa-aaaa-1.compute.amazonaws.com:8082 --agent-id ra8be27f45bd764a58 --agent-token d9417b45-2cd9-401b-927c-4d5c4912c614

   The following text displays in the Terminal window:

   Verifying archive integrity...  100%   All good.
   Uncompressing Lumada Data Catalog Agent Installer  100%
   
   
   This program installs Lumada Data Catalog Agent.
   
   Press ^C at any time to quit.
   
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                  LUMADA DATA CATALOG AGENT INSTALLER
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       1. Express Install          (Requires superuser access)
       2. Upgrade
       3. Exit
   
   Enter your choice [1-3]: 1
   Enter the name of the Lumada Data Catalog service user [ldcuser]:
   Enter install location [/opt/ldc] :
   Enter log location [/var/log/ldc] :
   Enter HIVE version [3.1.2]: 2.1.1
   Is Kerberos enabled? [y/N]: n
   ~~~~~~~~~~~~~~~~~~~~~~~
      SELECTION SUMMARY
   ~~~~~~~~~~~~~~~~~~~~~~~
   Lumada Data Catalog service user : ldcuser
                   Install location : /opt/ldc
                       Log location : /var/log/ldc
                   Kerberos enabled : false
   Proceed? [Y/n]:
   Directory /opt/ldc exists.
   Created directory /opt/ldc/agent
   Directory /var/log/ldc exists.
   Copying files ... done.
   
   Installed agent to /opt/ldc/agent
   Executing command: "/opt/ldc/agent/bin/agent" register --endpoint http://ec2-xx-xxx-xx-xx.aa-aaaa-1.compute.amazonaws.com:8082 --agent-token d9417b45-2cd9-401b-927c-4d5c4912c614 --agent-id ra8be27f45bd764a58 --no-exec false
   Registering agent
   done.

   The installation then continues in the browser at the public IP address for the node where Data Catalog was installed: http://<public_ip_address>:8082/setup

   NoteDo not log on to a fresh installation without running the Lumada Data Catalog installer setup process first.

Next steps

Parent Topic

• Downloading the Data Catalog packages