Post-install system configurations

Procedure

1. Point HiveServer2 to the location of the custom JAR files using one of the following three methods:

   • Place the Hive SerDe JAR and Data Catalog Hive format JAR in the hive/auxlib directory in the HiveServer2 node.
   • Copy the JARs to an alternate location on the HiveServer2 node and set the path of the alternate directory in the hive.aux.jars.path in hive-site.xml.

     For example:

     <property>
         <name>hive.aux.jars.path</name>
         <value>/usr/hdp/2.6.2.14-5/hive/lib</value>
         <description>
           The location of the plugin jars that contain implementations of the user defined functions and serdes.
         </description>
     </property>

   • Copy the JARs in the hive/auxlib directory or an alternate location on the HiveServer2 node and set the path of this directory as the HIVE_AUX_JARS_PATH in the hive-env.sh.

     For example:

     $ export HIVE_AUX_JARS_PATH=/usr/hdp/2.6.2.14-5/hive/lib

   NoteThe HiveServer2 and Data Catalog service user must have -rw-r--r-- permissions to these JARs. If the Data Catalog service user does not have ownership of this SerDe JAR, the format and schema discovery will not work.

2. Copy the ldc-hive-formats-6.1.0.jar file from the installation directory <Agent Location>/ext/ to the location you specified on the HiveServer2 node.

   For example:

   $ cp /opt/ldc/agent/ext/ ldc-hive-formats-6.1.0.jar\
        /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hive/auxlib/
   

3. (EMR only) Restart the Hive-Catalog service using the following commands:

   $ sudo stop hive-hcatalog-server
   $ sudo start hive-hcatalog-server

4. Restart HiveServer2 using the following commands:

   $ sudo stop hive-server2
   $ sudo start hive-server2

5. Copy the custom JAR files to the specified HDFS or S3 system directories.

   For example:

   <LDC Install Dir>$ cp agent/ext/ ldc-hive-formats-6.1.0.jar app-server/ext/
   <LDC Install Dir>$ cp agent/ext/ hive-serde-1.2.2.jar app-server/ext/<LDC Install Dir>$ 
   

6. Copy the custom JAR files to the Application Server directories in the <LDC Install Dir>/app-server/ext directory.

   For example:

   <LDC Install Dir>$ cp agent/ext/ ldc-hive-formats-6.1.0.jar app-server/ext/
   <LDC Install Dir>$ cp agent/ext/ hive-serde-1.2.2.jar app-server/ext/
   <LDC Install Dir>

7. Edit the Agent’s configuration.json file in the <LDC Install Dir>/agent/conf directory and the Application Server’s configuration.json file in the <LDC Install Dir>/app-server/conf directory with the following changes:

   1. Specify the location of the custom JAR files on the HDFS or S3 system.

   2. In the ldc.profile.customSerde.url key, specify the HDFS/SE path for the Hive SerDes as in the following example:

      " ldc.profile.customSerde.url " : {
      "value" : "hdfs://hdp.wld.com:8020/user/<yourcompany>/custom-jars",<path to custom SerDe jars>
      "type" : "STRING",
      "restartRequired” : true,
      "readonly" : true,
      "description" : "HDFS(hdfs:///) / S3(s3://) path containing custom serde jars to be added to spark sql at runtime", 
      "label" : "Custom Serde Jar Location",
      "category" : "MISC",
      "defaultValue" : "",
      "visible" : false,
      "uiConfig" : false
      },
      

8. Save and close the files.

Parent Topic

• Updating Hive Server with LDC JARs

If you have standalone agents and are using HDP 3.1, you must set up Hive and Spark to work together and enable Hive support in Data Catalog.

Parent Topic

• Configure CDH

Child Topics

• Enable Hive and Spark together in HDP 3.1
• Enable HDP 3.1 support

Parent Topic

• Configure an agent