Skip to main content
Version Pentaho
Hitachi Vantara Lumada and Pentaho Documentation

Packaging the Pentaho Worker Nodes Product

  1. Last updated
  2. Save as PDF

Before you install Pentaho Worker Nodes and HCI, consider how you want your system to operate. Below is an example configuration of the Pentaho Worker Nodes product on a typical customer site. Your IT Administrator will need to establish access between the Pentaho Server, the Pentaho Repository, and all product instances you want to run (worker nodes plus services), regardless of where each node is located. For example, in the same way that the Pentaho Server needs to talk to the Pentaho Repository, the worker node container requires the same ability. All the custom configurations you would apply to the Pentaho Server apply to the worker node container you are building. 

So, it is mandatory to configure your repository, and all the databases you might be using or want to use, in the worker node environment. Since the repository will be accepting multiple remote connections from the worker nodes and the Pentaho Server, you must configure the repository database to accept multiple remote connections. See Customize Worker Nodes for further information.

PWN_Scenario.png

Building the HCI Assembly Artifact

Before you can install and configure Worker Nodes, you must build the HCI Assembly Artifact.

The artifact must meet the following requirements:

  • Java JDK 1.8. Note: Later Java releases are not supported.
  • 64-bit Linux distribution
  • Docker version 1.10.3 or later

The product containing the necessary scripts to package the worker-node service and to package the final product can be downloaded from the Customer Support Portal under the name: pentaho-workernodes-8.0.0.0-XXX-dist.zip

Access to this ZIP file and the Pentaho Worker Nodes product is available to select customers. Contact your Hitachi Vantara sales associate for more information.  

Since the Pentaho Worker Nodes Product is a distribution package, follow the instructions below to unpack it.

  1. On the server where you want to deploy the Pentaho Worker Nodes Product, open a command line interface and navigate to the folder where the pentaho-workernodes-8.0.0.0-XXX-dist.zip is located.  For example:

cd /home/user/Downloads 

  1. Extract the  pentaho-workernodes-8.0.0.0-XXX-dist.zip file. 

unzip pentaho-workernodes-8.0.0.0-XXX-dist.zip

  1. Navigate to the workernodes folder that was created from the previous step.

cd  pentaho-workernodes-8.0.0.0-XXX

  1. Run the EULA installer.

java -DINSTALL_PATH=pentaho-worker-nodes -DEULA_ACCEPT=true -jar pentaho-workernodes-8.0.0.0.0-XXX/installer.jar -options-system 

  1. When complete, navigate to the INSTALL_PATH (pentaho-worker-nodes in the above example):

cd pentaho-worker-nodes

6.    Extract the worker nodes archive using the following command:

tar -zxf pentaho-workernodes-8.0.0.0.0-XXX.tar.gz

  1. Download the pentaho-server-8.0.0.0.zip file. 

File names and licenses are checked during the component build process. Missing or incorrectly named files will halt the build process. Be sure to check the accuracy of your files and licenses prior to downloading.

  1. Copy the  pentaho-server archive into the pentaho-ee-foundry/services/worker-node-service/docker folder.  For example:

cp ~/Downloads/pentaho-server-8.0.0.0.xxx.zip pentaho-ee-foundry/services/worker-node-service/docker 

Customize Worker Nodes

Before running the build scripts, you must add any customizations for the Worker Nodes, which can range from simple configuration file changes to complex custom plugins. For example, a simple configuration file change might include a debugging file that is required to have a worker node communicate with the repository you are using, while a more complex customization might include a custom PDI plugin that you want the worker node to be able to use. 

Add Custom Configuration Files and Plugins

Once you have planned your Worker Nodes setup, use the following guidelines to add custom configuration files and plugins.

It is recommended that you thoroughly check your current configuration. If you modify your configuration, you will need to repackage the Pentaho Worker Nodes Product.

Within the pentaho-ee-foundry/services/worker-node-service/docker directory add your custom configuration files and plugins in the following locations:

  • OSGi  Kettle and Platform Plugins: Drop any OSGi kettle plugins in the pentaho-server/pentaho-solutions/system/karaf/deploy directory.
  • Customized Configuration Files: Place customized configuration files in the same location on the directory structure as they are in the main pentaho-server directory structure. For example, the pentahoObjects.spring.xml file should go in the pentaho-server/pentaho-solutions/system/ directory just like in a default build of the Pentaho Server.
  • Kettle Legacy Plugins: Drop any legacy kettle plugins in the directory structure in the  pentaho-server/pentaho-solutions/system/kettle/plugins/steps directory.
  • Platform Legacy Plugins: Drop legacy platform plugins in the pentaho-server/pentaho-solutions/system/ directory.

Best Practices for Applying Customizations

It is recommended that you do not explicitly reference IP and port values in your XML files so you do not unintentionally bind your Docker images to these specific values. Otherwise, your Docker images would be unusable in different deployment environments that require different IPs and ports. For example, the IP and port you use to set up Worker Nodes for demonstration purposes differ from the IP and port you use in a production environment.

To avoid rebuilding and redeploying images when different IP and ports are required, you need to replace the explicit values in your repository.xml and postgres.hibernate.cfg.xml tokens as shown below.

You can replace these values with tokens as follows:

  1. In your repository.xml file, look for patterns such as the following:
jdbc:postgresql://10.0.51.5:5432/jackrabbit
  1. Replace the above pattern with the following pattern:
jdbc:postgresql://${repository.ip}:${repository.port}/jackrabbit
  1. In your postgres.hibernate.cfg.xml file, look for patterns such as the following:
jdbc:postgresql://10.0.51.5:5432/hibernate
  1. Replace the above pattern with the following pattern:
jdbc:postgresql://${repository.ip}:${repository.port}/hibernate
  1. Enter the explicit values in the Pentaho Server’s pentaho.worker.node.cfg file. Using the above example, the file would resemble the following:
wn-setup-environment-variables=\
REPOSITORY_IP=10.0.51.5,\
REPOSITORY_PORT=5432

Caution for Modifying Files with Mandatory Shipped Customizations 

While you are modifying files with your customizations, be aware that some files in the directory already contain required customizations. The following is a list of files and their shipped customizations which will need to be present after you complete your modifications:

  • pentaho-server/tomcat/webapps/pentaho/WEB-INF/web.xml
    • Line 27: <web-appmetadata-complete="true">
    • Line 29: <absolute-ordering/>
    • Line107: Proxy Trusting filter, add ${trust.ip}
  • pentaho-server/tomcat/webapps/pentaho/META-INF/context.xml
    • This file references Quartz by default. Make sure the Quartz database is removed or commented out in this file.
  • pentaho-server/pentaho-solutions/system/hibernate/postgresql.hibernate.cfg.xml. (Note that while this directory structure uses the Hibernate database, your database may be different.) 
    • Make sure to use ${repository.ip} and ${repository.port} instead of pointing to static db ip and port.
  • pentaho-server/pentaho-solutions/system/jackrabbit/repository.xml
    • Make sure to use ${repository.ip} and ${repository.port} instead of pointing to static db ip and port.
    • For the cluster journal, a configuration required to have more than one jackrabbit instance talking to the same database, make sure the Cluster ID has "${jackrabbit.cluster.node.id}"
    • Make sure that the AccessControlProvider points to org.apache.jackrabbit.core.security.authorization.acl.WorkerNodePentahoACLProvider
  • pentaho-server/pentaho-solutions/system/karaf/etc/org.apache.karaf.features.cfg
    • If you have any custom karaf features enabled, please make sure that featuresBoot contains pentaho-worker-nodes-ee.
  • pentaho-ee-foundry/services/worker-node-service/docker/licenses
    • To properly create Worker Nodes, place your license files in this directory. 

Licenses are a mandatory group of files that must be placed in the folder structure to properly create the Worker Nodes. See Download the Required Licenses for more information. 

Installing the Pentaho Worker Nodes Product

These instructions assume that you have built your HCI Assembly Artifact and customized your files for your deployment of Worker Nodes. When you are ready to create the final HCI product, start up the host machine where you want to deploy HCI.

There are several sections within these instructions, including:

Download the Pentaho Worker Nodes Product Files

When you install the Pentaho Server, the pentaho-workernodes-8.0.0.0-XXX-dist.zip is installed in the folder pentaho/* folder. 

  • Download the Pentaho Server 8.0.0.0 zip file and copy it into the pentaho-ee-foundry/services/worker-node-service/docker folder.

Download the Required Licenses

You will need to download the Pentaho Enterprise licenses used by your business and the Pentaho for HCI trial license.

  • Download and copy the pentaho-ee licenses into the /pentaho-ee-foundry/services/worker-node-service/docker/licenses folder.

For more information about how to download Pentaho licenses using the command line, see Manage Licenses using the Command Line Interface.

  • Navigate to the location where you unpacked the pentaho-workernodes-8.0.0.0.0-XXX.tar.gz file. Copy the pentaho-hci-trial-license.plk file into the /pentaho-ee-foundry/services/worker-node-service/docker/licenses folder.

We recommend you use the pentaho-hci-trial-license.plk license when deploying the product.  This license is only valid for using the Pentaho functionality within HCI and is valid for a period of one year.

Build Worker Nodes Service Components

At this point, apply your configuration changes as discussed in Customize Worker Nodes.

  1. Navigate to the pentaho-ee-foundry folder and run the ./create-pentaho-product-components.sh script.

If your site is behind a proxy firewall, you may need to adjust your docker file and component shell scripts to allow for passing a proxy value.

This script performs the following:

  • Creates the Docker images for Worker Nodes services and saves the image tar.gz file to the pentaho-ee-foundry/services/worker-node-service/service-components folder.
  • Under the service-components folder, creates the service plugin archive using the following three components: 
    • docker image
    • service plugin jar 
    • pkg tar.gz 
  1. Check that the service plugin archive is copied to the pentaho-ee-foundry/product-components folder.

Build Required Keys to Build the Worker Nodes Product

When you start HCI, you must specify a public/private key used to sign in and verify the update package. While there are multiple ways to generate the required keys, we suggest you use the GNU Privacy Guard (GPG) tool with the following steps.

  1. Using the tool GPG tool (also known as GnuPG), generate the keys. Follow the instruction and note the USER_NAME and password. 

gpg2 --gen-key

  1. Export the public key using the USER_NAME chosen during the key generation. This export will write the value into a file called public.key.

gpg2 --export -a "<USER_NAME>" > public.key

  1. Export the private key using the USER_NAME chosen during the key generation. This export will write the value into a file called private.key.

gpg2 --export-secret-keys -a "<USER_NAME>" > private.key

  1. Optionally, if the GPG2 --gen-key fails, follow these additional steps:
  1. Run the following command: 

sudo apt install  pinentry-qt

  1. Unset DISPLAY.
  2. Even though GPG2 instructs moving the mouse, typing commands, or doing something else with the system, the key generation steps can take a long time. To reduce this time, you can install the random number generation tool using the following command: 

sudo apt install rng-tools

Build the PentahoWorkerNodes product by executing the script./create-pentaho-product.sh. This script expects the public.key to be in the current folder. 

Install on a Single Instance of HCI

After packaging the Pentaho Worker Nodes Product, see Install Pentaho Worker Nodes on a Single Instance of HCI for an example of how to install your package on a single instance of HCI.