Packaging the Pentaho Worker Nodes Product
Before you install Pentaho Worker Nodes and HCI, consider how you want your system to operate. Below is an example configuration of the Pentaho Worker Nodes product on a typical customer site. Your IT Administrator will need to establish access between the Pentaho Server, the Pentaho Repository, and all product instances you want to run (worker nodes plus services), regardless of where each node is located. For example, in the same way that the Pentaho Server needs to talk to the Pentaho Repository, the worker node container requires the same ability. All the custom configurations you would apply to the Pentaho Server apply to the worker node container you are building.
So, it is mandatory to configure your repository, and all the databases you might be using or want to use, in the worker node environment. Since the repository will be accepting multiple remote connections from the worker nodes and the Pentaho Server, you must configure the repository database to accept multiple remote connections. See Customize Worker Nodes for further information.
Building the HCI Assembly Artifact
Before you can install and configure Worker Nodes, you must build the HCI Assembly Artifact.
The artifact must meet the following requirements:
- Java JDK 1.8. Note: Later Java releases are not supported.
- 64-bit Linux distribution
- Docker version 17.05.0-ce or later
The product containing the necessary scripts to package the worker-node service and to package the final product can be downloaded from the Customer Support Portal under the name:
Access to this ZIP file and the Pentaho Worker Nodes product is available to select customers. Contact your Hitachi Vantara sales associate for more information.
Since the Pentaho Worker Nodes Product is a distribution package, follow the instructions below to unpack it.
- On the server where you want to deploy the Pentaho Worker Nodes Product, open a command line interface and navigate to the folder where the
pentaho-workernodes-184.108.40.206-XXX-dist.zipis located. For example:
- Extract the
- Navigate to the
workernodesfolder that was created from the previous step.
- Run the EULA installer.
java -DINSTALL_PATH=pentaho-worker-nodes -DEULA_ACCEPT=true -jar pentaho-workernodes-220.127.116.11.0-XXX/installer.jar -options-system
- When complete, navigate to the INSTALL_PATH (pentaho-worker-nodes in the above example):
6. Extract the worker nodes archive using the following command:
tar -zxf pentaho-workernodes-18.104.22.168.0-XXX.tar.gz
- Download the
File names and licenses are checked during the component build process. Missing or incorrectly named files will halt the build process. Be sure to check the accuracy of your files and licenses prior to downloading.
- Copy the
pdi-ee-clientarchive into the
pentaho-ee-foundry/jobs/pdi-job-plugin/docker/directory. For example:
Customize Worker Nodes
Before running the build scripts, you must add any customizations for the Worker Nodes, which can range from simple configuration file changes to complex custom plugins. For example, a simple configuration file change might include a debugging file that is required to have a worker node communicate with the repository you are using, while a more complex customization might include a custom PDI plugin that you want the worker node to be able to use.
Add Custom Configuration Files and Plugins
Once you have planned your Worker Nodes setup, use the following guidelines to add custom configuration files and plugins.
It is recommended that you thoroughly check your current configuration. If you modify your configuration, you will need to repackage the Pentaho Worker Nodes Product.
pentaho-ee-foundry/jobs/pdi-job-plugin/docker/directory add your custom configuration files and plugins in the following locations:
- OSGi Kettle and Platform Plugins: Drop any OSGi kettle plugins in the
- Customized Configuration Files: Place customized configuration files in the same location on the directory structure as they are in the main
data-integrationdirectory structure. For example:
log4j.xmlfile should go under
data-integration/classes, just like in a default build of Pentaho Data Integration (PDI).
repositories.xmlfiles should go under
.kettle, just like in a default build of PDI.
- Legacy Kettle Plugins and Kettle Step Plugins: Drop any legacy kettle plugins and kettle step plugins in the directory structure in the
Best Practices for Applying Customizations
It is recommended that you do not explicitly reference IP and port values in your XML files so you do not unintentionally bind your Docker images to these specific values. Otherwise, your Docker images would be unusable in different deployment environments that require different IPs and ports. For example, the IP and port you use to set up Worker Nodes for demonstration purposes differs from the IP and port you use in a production environment.
Set a Repository Definition
To avoid rebuilding and redeploying images when different IP and ports are required, set the following explicit repository definition in your
repositories.xml file as follows:
<repository> <id>PentahoEnterpriseRepository</id> <name>pentaho-repo</name> <description>Pentaho repository</description> <is_default>true</is_default> <repository_location_url>http://REPO_IP:REPO_PORT/pentaho</repository_location_url> <version_comment_mandatory>N</version_comment_mandatory> </repository>
In the above code definition, "pentaho-repo", "REPO_IP" and "REPO_PORT" are keys managed by the Pentaho Worker Nodes. Be sure to set these keys as-is.
Licenses are a mandatory group of files that must be placed in the folder structure to properly create the Worker Nodes. See Download the Required Licenses for more information.
Building the Pentaho Worker Nodes Product File
To build the Pentaho Worker Nodes product, you will need both the HCI Assembly Artifact and Package-Tool files, and the Pentaho license files. The instructions for downloading the license files are below. The HCI Assembly Artifact and Package-Tool files are the property of HCI. Contact your Pentaho Enterprise Sales representative to obtain these files.
There are several sections within these instructions, including:
- Download Required Licenses
- Build Worker Nodes Service Components
- Build Required Keys to Build the Worker Nodes Product
- Add the HCI Assembly Artifact and Package-Tool Files
- Build the Pentaho Worker Nodes Product File
Download the Required Licenses
You will need to download the Pentaho Enterprise licenses used by your business and the Pentaho for HCI trial license.
- Download and copy the pentaho-ee licenses into the
For more information about how to download Pentaho licenses using the command line, see Manage Licenses using the Command Line Interface.
- Navigate to the location where you unpacked the
pentaho-workernodes-22.214.171.124.0-XXX.tar.gzfile. Copy the
pentaho-hci-trial-license.plkfile into the
We recommend you use the
pentaho-hci-trial-license.plk license when deploying the product. This license is only valid for using the Pentaho functionality within HCI and is valid for a period of one year.
Build Worker Nodes Service Components
At this point, apply your configuration changes as discussed in Customize Worker Nodes.
- Navigate to the
pentaho-ee-foundryfolder and run the
If your site is behind a proxy firewall, you may need to adjust your docker file and component shell scripts to allow for passing a proxy value. If this situation applies to you, see Troubleshooting: Adjust the Docker File and Component Shell Scripts to Pass Proxy Values.
This script performs the following:
- Creates the Docker images for Worker Nodes job and saves the
image tar.gzfile to the
- Creates the Docker images for Content Execution Broker services and saves the
image tar.gzfile to the
- Under the
service-componentsfolder, creates the service plugin archive using the following three components:
- docker image
- service plugin jar
- Under the
job-componentsfolder, creates the job plugin archive using the following three components:
- docker image
- job plugin jar
- Check that the service plugin archive and the job plugin archive are copied to the
Build Required Keys to Build the Worker Nodes Product
When you start HCI, you must specify a public/private key used to sign in and verify the update package. While there are multiple ways to generate the required keys, we suggest you use the GNU Privacy Guard (GPG) tool with the following steps.
- Using the tool GPG tool (also known as GnuPG), generate the keys. Follow the instruction and note the USER_NAME and password.
- Export the public key using the USER_NAME chosen during the key generation. This export will pipe the value into a file called public.key.
gpg2 --export -a "<USER_NAME>" > public.key
- Export the private key using the USER_NAME chosen during the key generation. This export will pipe the value into a file called private.key.
gpg2 --export-secret-keys -a "<USER_NAME>" > private.key
- Optionally, if the GPG2 --gen-key fails, follow these additional steps:
- Run the following command:
sudo apt install pinentry-qt
- Unset DISPLAY.
- Even though GPG2 instructs moving the mouse, typing commands, or doing something else with the system, the key generation steps can take a long time. To reduce this time, you can install the random number generation tool using the following command:
sudo apt install rng-tools
- Copy the newly created
private.keyfiles to the the
Add the HCI Assembly Artifact and Package-Tool Files
In this task, you will add the HCI and Package-Tool files to your Pentaho worker nodes build directories.
This task assumes you have acquired the HCI Assembly Artifact and Package-Tool files from your Pentaho Enterprise Sales representative.
- Copy the the
package-tool-126.96.36.199.tgzfiles into the
pentaho-ee-foundry/directory prior to packaging.
- Create a directory called 'package-tool' under the
- Unpack the Package-tool archive into the newly created package-tool directory.
Build the Pentaho Worker Nodes File
You will now build the
PentahoWorkerNodes product by executing the script
./create-pentaho-product.sh. This script expects the
public.key to be in the correct folder.
- Check that the public.key is in the current
- From the same directory, run the script
The execution of this script will build the Pentaho Worker Nodes Product.
Install on a Single Instance of HCI
After packaging the Pentaho Worker Nodes Product, see Install Pentaho Worker Nodes on a Single Instance of HCI for an example of how to install your package on a single instance of HCI.