Set Up Pentaho to Connect to a Hortonworks Cluster

Verify or Edit config.properties (Unsecured Clusters)

If you are connecting to an unsecure cluster, verify that these values are properly set.  Set the Oozie proxy user if needed. 

1. Navigate to the pentaho-big-data-plugin/hadoop-configurations/hdpxx directory and open config.properties.
2. Add the following values:

3. Save and close the file.

Edit config.properties (Secured Clusters)

If you are connecting to a secure cluster, add Kerberos information to the config.properties file.  Set the Oozie proxy user if needed.

1. Navigate to the pentaho-big-data-plugin/hadoop-configurations/hdpxx directory and open the config.properties file.
2. Add these values:

3. Save and close the file.

Edit hbase-site.xml

Edit the location of the temporary directory in the hbase-site.xml file to create an HBase local storage directory.

1. Navigate to the pentaho-big-data-plugin/hadoop-configurations/hdpxx directory and open the hbase-site.xml file.
2. Add the following value:

3. Save and close the file.

Edit hive-site.xml

Verify that the following parameter is set in the hive-site.xml file:

1. Navigate to the pentaho-big-data-plugin/hadoop-configurations/hdpxx directory and open the hive-site.xml file.
2. Add the following value:

3. Save and close the file.

Edit mapred-site.xml

Edit the mapred-site.xml file to indicate where the job history logs are stored and to allow MapReduce jobs to run across platforms. 

1. Navigate to the pentaho-big-data-plugin/hadoop-configurations/hdpxx directory and open the  mapred-site.xml file .
2. Add the following values:

<property>
	<name>mapreduce.application.classpath</name>
	<value>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*
			:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*
			:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*
			:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*
			:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*
			:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure
	</value>
</property>

<property>
  <name>mapreduce.application.framework.path</name>
  <value>/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework</value>
</property>

<property>
  <name>mapreduce.app-submission.cross-platform</name>
  <value>true</value>
</property>

<property> <name>yarn.application.classpath</name>
 <value>$HADOOP_CONF_DIR,/usr/hdp/current/hadoop-client/*,
/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,
/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,
/usr/hdp/current/hadoop-yarn-client/lib/*</value>
 </property>

Set the Active Shim in Spoon

Set the active shim when you want to connect to a Hadoop cluster the first time, or when you want to switch clusters.  To set a shim as active, complete the following steps:

1. Start Spoon.
2. Select Hadoop Distribution... from the Tools menu.

3. In the Hadoop Distribution window, select the Hadoop distribution you want.
4. Click OK.
5. Stop, then restart Spoon.

Configure and Test the Cluster Connection

You must provide connection details for the cluster and services you will use, such as the hostname for HDFS or the URL for Oozie.  Then, you can use a built-in tool to test your configuration to find and troubleshoot common configuration issues, such as wrong hostnames and user permission errors.

Connection settings are set in the Hadoop cluster window.  You can get to the settings from several places, but in these instructions, you will get the Hadoop cluster window from the View tab in a transformation or job. Complete the following steps to configure and test a connection:

1. In Spoon, create a new job or transformation or open an existing one.
2. Click the View tab.

3. Right-click the Hadoop cluster folder, then click New.  The Hadoop cluster window appears.  

4. Enter information in the Hadoop cluster window.  You can get this information from your Hadoop Administrator.

As a best practice, use Kettle variables for each connection parameter value to mitigate risks associated with running jobs and transformations in environments that are disconnected from the repository. 

5. Click the Test button.  Test results appear in the Hadoop Cluster Test window.  If you have errors, see the Troubleshoot Cluster and Service Configuration Issues section below to resolve the issues, then test again.

6. Click Close on the Hadoop Cluster Test window, then click OK to close the Hadoop cluster window.

Copy Spoon Shim Files to Other Pentaho Components

Once your connection has been properly configured on Spoon, copy configuration files to the shim directories in other Pentaho components. Copy the following configuration files from the pentaho-big-data-plugin/hadoop-configurations/hdpxx directory to pentaho-big-data-shim/hdpxx on the Pentaho Server, PRD, or PME:

• hbase-site.xml
• core-site.xml
• hdfs-site.xml
• hive-site.xml
• mapred-site.xml
• yarn-site.xml

Set the Active Shim on PRD, PME, and the Pentaho Server

Modify the plugin.properties file to set the active shim for the Pentaho Server, PRD, and PME.

1. Stop the component.
2. Locate the pentaho-big-data-plugin directory for your component. 
3. Navigate to the hadoop-configurations directory.
4. Navigate to the pentaho-big-data-plugin directory and open the plugin.properties file.
5. Set the active.hadoop.configuration property to the directory name of the shim you want to make active.  Here is an example:

active.hadoop.configuation=hdp24

6. Save and close the plugin.properties file.
7. Restart the component.

Create and Test Connections

Connection tests appear in the following table.

Once you've connected to the cluster and its services properly, provide connection information to users who need access to the cluster and its services.  Those users can only obtain access from computers that have been properly configured to connect to the cluster.

Here is what they need to connect:

• Hadoop Distribution and version of the cluster
• HDFS, JobTracker, ZooKeeper, and Hive2/Impala Hostnames, IP addresses and port numbers
• Oozie URL (if used)
• Users also require the appropriate permissions to access the directories they need on HDFS.  This typically includes their home directory and any other required directories.

They might also need more information depending on the job entries, transformation steps, and services they use.  Here's a more detailed list of information that your users might need from you.

Sqoop Support

If you receive an error message stating Generating splits for a textual index column allowed only in case of "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" property passed as a parameter while trying to use the split-by option to the Sqoop Import job entry with the HDP 2.5 shim, perform the following steps to set the org.apache.sqoop.splitter.allow_text_splitter property to true:

1. Open your KJB file that contains a Sqoop Import entry in the PDI client (also known as Spoon).

2. Double-click the Sqoop Import entry to access the Sqoop Import property dialog box.

3. Click the Advanced Options link in the lower left corner of the dialog box.

4. In the Custom tab, add the Dorg.apache.sqoop.splitter.allow_text_splitter argument and set the value to true.

5. Click OK and save your KJB file.

You should now be able to use the split-by option to the Sqoop Import entry.

Simba Spark SQL Driver Support

If you are using Pentaho 7.0 or later, the HDP 2.4 shim supports the Simba Spark SQL driver. You will need to download, install, and configure the driver to use Simba Spark SQL with the HDP 2.4 shim.

1. Download the Simba Spark SQL driver.
2. Extract the ZIP file, and then copy the following 3 files into the lib/ directory of the HDP shim:

• SparkJDBC41.jar
• TCLIServiceClient.jar
• QI.jar

3. In the Database Connection window, select SparkSQL option. The default port for the Spark thrift server is 10015.

4. For secure connections, set the following additional parameters on the JDBC URL through the Options tab:

• KrbServiceName
• KrbHostFQDN
• KrbRealm

5. For unsecure connections, if your Spark SQL configuration specifies hive.server2.authentication=NONE, then make sure to include an appropriate User Name in the Database Connection window.  Otherwise, the connection is assumed to be NOSASL authentication, which will cause a connection failure after timeout.

6. Stop and restart the component.

Pentaho can connect to HDP 2.3 cluster using the HDP 2.2 or HDP 2.3 shims

You can use either the HDP 2.2 or HDP 2.3 shims to connect to a HDP 2.3 clusters:

• If you use the HDP 2.2 shim to connect to an HDP 2.3 cluster, only HDP 2.2 functionality is supported. 
• If you want to support HDP 2.3 functionality, use the HDP 2.3 shim to connect to the HDP 2.3 cluster instead.

Shims can be downloaded from the Pentaho Customer Support Portal.