Big Data issues
Follow the suggestions in these topics to help resolve common issues when working with Big Data:
- General configuration problems
- Cannot access cluster with Kerberos enabled
- Cannot access the Hive service on a cluster
- HBase Get Master Failed error
- Sqoop import into Hive fails
- Pig job not executing after Kerberos authentication fails
- Kettle cluster on YARN will not start
- Group By step is not supported in a single threaded transformation engine
- Hadoop on Windows
- Spark issues
- Legacy mode activated when named cluster configuration cannot be located
- Unable to read or write files to HDFS on the Amazon EMR cluster
- Use YARN with S3
- Lumada Data Catalog searches returning incomplete or missing data
See Pentaho Troubleshooting articles for additional topics.
General configuration problems
The issues in this section explain how to resolve common configuration problems.
When updating to Pentaho 9.0, you are required to perform a one-time operation to update your cluster configurations to use multiple cluster features. For more information, see Set up the Pentaho Server to connect to a Hadoop cluster. If you do not update your cluster configurations, then the legacy configuration from the Big Data plugin is used.
Driver and configuration issues
Symptoms | Common Causes | Common Resolutions |
Could not find cluster configuration file
config.properties for the cluster in expected metastore locations
or a legacy shim configuration. |
|
|
Could not find service for interface associated with named cluster. |
|
|
No driver. |
|
|
Driver does not load. |
|
|
The file system's URL does not match the URL in the configuration file. |
|
|
Sqoop Unsupported major.minor version Error. | In Pentaho 6.0, the Java version on your cluster is older than the Java version that Pentaho uses. |
|
Connection problems
Symptoms | Common Causes | Common Resolutions |
Hostname does not resolve. |
|
|
Port number does not resolve. |
|
|
Cannot connect to the cluster. |
|
|
Windows failure message: “java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset |
| Please follow the instructions at https://cwiki.apache.org/confluence/display/HADOOP2/WindowsProblems and set the environment variable %HADOOP_HOME% to point to the directory path containing WINUTILS.EXE. |
Cannot access a Hive database (Secured Clusters Only). |
|
|
Directory access or permissions issues
Symptoms | Common Causes | Common Resolutions |
Access error when trying to reach the User Home Directory. |
|
|
Cannot access directory. |
|
|
Cannot create, read, update, or delete files or directories. |
|
|
Test file cannot be overwritten. |
|
|
Oozie issues
Symptoms | Common Causes | Common Resolutions |
Cannot connect to Oozie |
|
|
Zookeeper problems
Symptoms | Common Causes | Common Resolutions |
Cannot connect to Zookeeper |
|
|
Zookeeper hostname or port not found or does not resolve properly |
|
|
Kafka problems
Symptoms | Common Causes | Common Resolutions |
Cannnot connect to Kafka |
|
|
Cannot access cluster with Kerberos enabled
If this issue persists, verify that the username, password, UID, and GID for each impersonated or spoofed user is the same on each node. When a user is deleted and recreated, it may then have different UIDs and GIDs causing this issue.
Cannot access the Hive service on a cluster
If you cannot use Kerberos impersonation to authenticate and access the Hive service on a cluster, review the steps in Set Up Kerberos for Pentaho.
If this issue persists, copy the hive-site.xml file on the Hive server to the configuration directory of the named cluster connection in these directories:
Pentaho Server
pentaho-server/pentaho-solutions/system/kettle/plugins/pentaho-big-data-plugin/hadoop-configurations/[cluster distribution]
PDI client
data-integration/plugins/pentaho-big-data-plugin/hadoop-configurations/[cluster distribution]
If the problem continues to persist, disable pooled connections for Hive.
HBase Get Master Failed error
- Pentaho Server:
pentaho-server/pentaho-solutions/system/kettle/plugins/pentaho-big-data-plugin/hadoop-configurations/[cluster distribution]
- PDI client:
data-integration/plugins/pentaho-big-data-plugin/hadoop-configurations/[cluster distribution]
Sqoop export fails
If executing a Sqoop export job and the system generates the following error because a file already exists at the destination, then Sqoop failed to clear the compile directory:
Could not rename
\tmp\sqoop-devuser\compile\1894e2403c37a663c12c752ab11d8e6a\aggregatehdfs.java to
C:\Builds\pdi-ee-client-9.0.0.0-MS-550\data-integration\.\aggregatehdfs.java. Error:
Destination 'C:\Builds\pdi-ee-client-9.0.0.0-MS-550\data-integration\.\aggregatehdfs.java'
already exists.
Despite the error message, the job that generated it ended successfully. To stop this error message, you can add a Delete step to the job to remove the compile directory before execution of the Sqoop export step.
Sqoop import into Hive fails
Verify the Hadoop connection information used by the local Hive installation is configured the same as the Sqoop job entry.
Pig job not executing after Kerberos authentication fails
For authentication with Pig, Pentaho uses the
UserGroupInformation
wrapper around a JAAS Subject with
username and password which is used for impersonation. The
UserGroupInformation
wrapper is stored in the
KMSClientProvider constructor. When the Kerberos ticket
expires, a new UserGroupInformation
is created, but the instance
stored in the KMSClientProvider constructor does not update.
The Pig job fails when Pig cannot obtain delegation tokens to authenticate the job
at execution time.
To resolve, set the key.provider.cache.expiry time to a value equal to or less than the duration time of the Kerberos ticket. By default, the key.provider.cache.expiry time is set to a value of: 10 days
Procedure
Navigate to the hdfs-site.xml file location.
- In the PDI client, navigate to: data-integration\plugins\pentaho-big-data-plugin\hadoop-configurations\hdp25
- For the Pentaho Server, navigate to: pentaho-server\pentaho-solutions\system\kettle\plugins\pentaho-big-data-plugin\hadoop-configurations\hdp25
Open the hdfs-site.xml file in a text editor.
Adjust the key.provider.cache.expiry value (in milliseconds) so that it is less than the duration time of the Kerberos ticket.
NoteYou can view the Kerberos ticket duration time in the krb5.conf file.<property> <name>dfs.client.key.provider.cache.expiry</name> <value>410000</value> </property>
Group By step is not supported in a single threaded transformation engine
- An entire set of rows sharing the same grouping key are filtered from the transformation before the Group By step.
- The Reduce single threaded option in the Pentaho MapReduce entry's Reducer tab is selected.
To fix this issue, open the Pentaho MapReduce entry and deselect the Reduce single threaded option in the Reducer tab.
Kettle cluster on YARN will not start
Verify in the File System Path (in the Files tab) that the Default FS setting matches the configured hostname for the HDFS Name node, then try starting the kettle cluster again.
Hadoop on Windows
If you are using Hadoop on Windows, you may get an "unexpected error" message. This message indicates that multiple cluster support across different versions of Hadoop is not available on Windows.
You are limited to using the same version of Hadoop for multiple cluster use on Windows. If you have problems accessing the Hadoop file system on a Windows machine, see the Problems running Hadoop on Windows article on the Hadoop Wiki site.
Spark issues
Follow the suggestions in these topics to help resolve common issues with running transformations with Spark.
Steps cannot run in parallel
Some steps cannot run in parallel (on multiple nodes in a cluster), and will produce unexpected results. However, these steps can run as a coalesced dataset on a single node in a cluster. To enable a step to run as a coalesced dataset, add the step ID as a property value in the configuration file for using the Spark engine.
Get the step ID
Each PDI step has a step ID, a globally unique identifier of the step. Use either of the following two methods to get the ID of a step:
Method 1: Retrieve the ID from the log
Procedure
In the PDI client, create a new transformation and add the step to the transformation. For example, if you needed to know the ID for the Select values step, you would add that step to the new transformation.
Set the log level to debug.
Execute the transformation using the Spark engine.
The step ID will display in the Logging tab of the Execution Results pane. For example, the log will display:Selected the SelectValues step to run in parallel as a GenericSparkOperation,
where SelectValues is the step ID.
Method 2: Retrieve the ID from the PDI plugin registry
If you have created your own PDI transformation step plugin, the step ID is one of the annotation attributes that the developer supplies.
Add the step ID to the configuration file
Perform the following steps to add another step ID to the configuration file:
Procedure
Navigate to the data-integration/system/karaf/etc folder and open the org.pentaho.pdi.engine.spark.cfg file.
Append your step ID to the forceCoalesceSteps property value list, using a pipe character separator between the step IDs.
Save and close the file.
Table Input step fails
If you run a transform using the Table Input step with a large database, the step will not complete. Use one of the following methods to resolve the issue:
Method 1: Load the data to HDFS before running the transform
Run a different transformation using the Pentaho engine to move the data to the HDFS cluster.
Then use HDFS Input to run the transformation using the Spark engine.
Method 2: Increase the driver side memory configuration
Navigate to the data-integration/adaptive-execution/config folder and open the application.properties file.
Increase the value of the sparkDriverMemory parameter, then save and close the file.
User ID below minimum allowed
To resolve, change the ID of the proxy user to be higher than the minimum user ID specified for the cluster.
Legacy mode activated when named cluster configuration cannot be located
If you run a transformation or job for which PDI cannot locate and load a named configuration cluster, then PDI activates a legacy mode. This legacy, or fallback, mode is only available in Pentaho 9.0 and later.
When the legacy mode is activated, PDI attempts to run the transformation by finding any existing cluster configuration you have set up in the PDI Big Data plugin. PDI then migrates the existing configuration to the latest PDI instance that you are currently running.
The legacy mode is helpful for transformations built with previous versions of PDI and includes individual steps that are not associated to a named cluster. You can run the transformation in legacy mode without revising the cluster configuration in each individual step. For information about setting up a named cluster, see Connecting to a Hadoop cluster with the PDI client.
When legacy mode is active, the transformation log displays the following message:
Could not find cluster configuration file {0} for cluster {1} in expected metastore
locations or a legacy shim configuration.
If the Big Data plugin is present and PDI accesses it to successfully activate legacy mode, the transfomation log displays the following message:
Cluster configuration not found in expected location; trying legacy
configuration location.
For more information about working with clusters, see Get started with Hadoop and PDI.
Unable to read or write files to HDFS on the Amazon EMR cluster
When running a transformation on an EMR cluster, the transformation appears to run successfully, but an empty file is written to the cluster. When PDI is not installed on the Amazon EC2 instance where you are running your transformation, you are unable to read or write files to the HDFS cluster. Any files written to the cluster are empty.
To resolve this issue, perform the following steps to edit the hdfs-site.xml file on the PDI client
:Procedure
Navigate to the <username>/.pentaho/metastore/pentaho/NamedCluster/Configs/<user-defined connection name> directory.
Open the hdfs-site.xml file with any text editor.
Add the following code:
<property> <name>dfs.client.use.datanode.hostname</name> <value>true</value> </property>
Save and close the file.
Use YARN with S3
When using the Start a PDI cluster on YARN and Stop a PDI cluster on YARN job entries to run a transformation that attempts to read data from an Amazon S3 bucket, the transformation fails. The transformation fails because the Pentaho metastore is not accessible to PDI on the cluster. To resolve this problem, verify that the Pentaho metastore is accessible to PDI on the cluster.
Perform the following steps to make the Pentaho metastore accessible to PDI:
Procedure
Navigate to the <user>/.pentaho/metastore directory on the machine with the PDI client.
On the cluster where the Yarn server is located, create a new directory in the design-tools/data-integration/plugins/pentaho-big-data-plugin directory, then copy the metastore directory into this location. This directory is the <NEW_META_FOLDER_LOCATION> variable.
Navigate to the design-tools/data-integration directory and open the carte.sh file with any text editor.
Add the following code in the line before the
export OPT
line:OPT="$OPT -DPENTAHO_METASTORE_FOLDER=<NEW_META_FOLDER_LOCATION>"
, then save and close the file.Create a zip file containing the contents of the data-integration directory.
In your Start a PDI cluster on YARN job entry, go to the Files tab of the Properties window, then locate the PDI Client Archive field. Enter the filepath for the zip file.
Results
- Avro Input
- Avro Output
- Orc Input
- Orc Output
- Parquet Input
- Parquet Output
- Text File Input
- Text File Output
Lumada Data Catalog searches returning incomplete or missing data
If you have a transformation that contains the Catalog Input, Catalog Output, Read Metadata, or Write Metadata steps, there may be instances when a complete search of the records in Lumada Data Catalog (LDC) is not performed. This error can occur if:
The default limit provided to prevent PDI from exceeding memory limits or stop connection timeouts to LDC is too short for your environment.
To resolve this issue:
- Design your transformation.
- Right-click on the canvas to open the Transformation properties dialog box.
- In the Parameters tab, add the following parameter:
catalog-result-limit
- In the Default Value column, enter a number greater than the
default value of
25
, for example500
. - Run your transformation.