Set Up the Adaptive Execution Layer (AEL)

Pentaho 8.1 Installation

When you install the Pentaho Server, the AEL daemon is installed in the folder data-integration/adaptive-execution. This folder will be referred to as 'PDI_AEL_DAEMON_HOME'.

Spark Client

The Spark client is required for the operation of the AEL daemon. The recommended versions of the Apache Spark client are 2.0, 2.1, and 2.2. Perform the following steps to install the Spark client.

1. Download the Spark client, spark-2.1.0-bin-hadoop2.7.tgz, from http://spark.apache.org/downloads.html.
2. Extract it to a folder on the cluster where the daemon can access it. This folder will be referred to as the variable 'SPARK_HOME'. 

Pentaho Spark Application

The Pentaho Spark application is built upon PDI's Kettle engine, which allows transformations to run unaltered within a Hadoop cluster. Some third-party plugins, such as those plugins available in the Pentaho Marketplace, may not be included by default within the Pentaho Spark application. To address this issue, we include the Spark Application builder tool so you can customize the Pentaho Spark application by adding or removing components to fit your needs. 

After running the Spark application builder tool, copy and unzip the resulting pdi-spark-driver.zip file to an edge node in your Hadoop cluster. The unpacked contents consist of the data-integration folder and the pdi-spark-executor.zip file, which includes only the required libraries needed by the Spark nodes themselves to execute a transformation when the AEL daemon is configured to run in YARN mode. Since the pdi-spark-executor.zip file needs to be accessible by all nodes in the cluster, it must be copied into HDFS. Spark distributes this .zip file to other nodes and then automatically extracts it.

Perform the following steps to run the Spark application build tool and manage the resulting files.

1. Ensure that you have configured your PDI client with all the plugins that you will use.
2. Navigate to the design-tools/data-integration folder and locate the spark-app-builder.bat (Windows) or the spark-app-builder.sh (Linux).
3. Execute the Spark application builder tool script. A console window will display and the pdi-spark-driver.zip file will be created in the data-integration folder (unless otherwise specified by the -outputLocation parameter described below). 

   The following parameters can be used when running the script to build the pdi-spark-driver.zip.

   Parameter	Action
   –h or --help	Displays the help.
   –e or --exclude-plugins	Specifies plugins from the data-integration/plugins folder not to exclude from the assembly.
   –o or --outputLocation	Specifies the output location.

    

4. The pdi-spark-driver.zip file contains a data-integration folder and pdi-spark-executor.zip file. Copy the data-integration folder to the edge node where you want to run the AEL daemon.  
5. Copy the pdi-spark-executor.zip file to the HDFS node where you will run Spark. This folder will be referred to as 'HDFS_SPARK_EXECUTOR_LOCATION'.   

For the cluster nodes to use the functionality provided by PDI plugins when executing a transformation, they must be installed into the PDI client prior to generating the Pentaho Spark application. If you install other plugins later, you must regenerate the Pentaho Spark application.

Cloudera

If your Cloudera Spark client does not contain the Hadoop libraries, you must add the Hadoop libraries to the classpath using the SPARK_DIST_CLASSPATH environment variable. To do this they can use the following command:

export SPARK_DIST_CLASSPATH=$(hadoop classpath)

Hortonworks

The Hortonworks Data Platform (HDP) version on your edge node where your Pentaho server resides must be the same version used on your cluster or the AEL daemon and the PDI client will stop working. To prevent this from happening, you must export the HDP_VERSION variable. For example:

export HDP_VERSION=${HDP_VERSION:-2.6.0.3-8}

You can check the HDP version on your cluster with the following command:

hdp-select status hadoop-client

MapR 

To configure the AEL daemon to run in a MapR Spark 2.1 production environment, complete the following steps.

1. Navigate to the .../data-integration/adaptive-execution/config directory and open the application.properties file.
2. Set the following property for your MapR Spark 2.1 environment:

   Property	Value
   hadoopConfDir	This property identifies the Hadoop cluster that Spark will use.  Because MapR identifies the Hadoop cluster by default, set the property value to empty, as shown here: hadoopConfDir=""
   -Dhadoop.login	This property identifies the security environment that the Hadoop cluster will use. If you enable security, the value of the MAPR_ECOSYSTEM_LOGIN_OPTS environment variable will include the 'hybrid JVM' option for the hadoop.login property. Set the property value to 'hybrid'  to specify a mixed security environment using Kerberos and internal MapR security technologies as shown here: -Dhadoop.login=hybrid
   -Djava.security.auth.login.config	This property identifies the configuration file to use when you enable security. The MapR distribution for Hadoop uses the Java Authentication and Authorization Service (JAAS) to control security features. The /opt/mapr/conf/mapr.login.conf file specifies configuration parameters for JAAS. Set the property value to /opt/mapr/conf/mapr.login.conf as shown here: -Djava.security.auth.login.config=/opt/mapr/conf/mapr.login.conf

    

3. Save and close the file.
4. Before running the daemon, add the Hadoop libraries to the classpath by running the following command from the command prompt (terminal window) on the cluster:

   export SPARK_DIST_CLASSPATH=$(hadoop classpath)

 You can now test your AEL configuration by creating a run configuration using the Spark engine. Refer to Run Configurations for more details. 

Amazon EMR

When running AEL on Amazon EMR, LZO compression and Oracle JDK 8 are required components.