Skip to main content

Pentaho+ documentation has moved!

The new product documentation portal is here. Check it out now at docs.hitachivantara.com

 

Hitachi Vantara Lumada and Pentaho Documentation

Setting up mounted volumes on Pentaho Worker Nodes

Parent article

You can use mounted volumes to manage your Pentaho Worker Nodes configuration and expanding default capabilities. Mounted volumes are a native Docker capability. For more information about mounted volumes, see the Docker documentation on Volumes.

Pentaho Worker Nodes leverage mounted volumes capability for the following tasks:

  • Adding or updating expired Pentaho licenses.
  • Adding a new PDI step plugin.
  • Adding custom JAR files.
  • Changing the details of a Pentaho Repository connection.

The files in the mounted volume have the following structure:

<some-dir>
  |--- .kettle ( <dot><kettle> )
    |--- custom-lib 
    |--- data 
    |--- licenses 
    |--- plugins 
    |--- plugins-osgi 

Creating the folder structure

You can create the folder structure by running a utility script located in the Pentaho Worker Nodes installation folder.

Procedure

  1. In your Pentaho Worker Nodes installation directory, navigate to <product-install_dir>/jobs/com.pentaho.foundry.plugins.job.pdi/<pdi_job_version>/package/svcbin/util/create-kettle-volume-dir.sh.

  2. Run the script: ./create-kettle-volume-dir.sh<path-to-some-dir> to create the folder structure under the <path-to-some-dir>.

  3. Move the following files to the specifed folder in the folder structure:

    File(s)Folder location
    kettle.properties, repositories.xml, jdbc.properties, and log4j.xml files.kettle
    Custom JAR files for PDI runtimecustom-lib
    input files used in KTR and KJB execution data
    .installedLicenses.xmllicenses
    PDI pluginsplugins
    PDI OSGI plugins or features.cfg files plugins-osgi

Required content in mounted volume

There is a minimum set of required content that you must provide for Pentaho Worker Nodes to operate properly. Use the table below to place the specified files properly in the mounted volume directory.

FileFolder locationDescription
repositories.xml.kettleThis file contains the details of the Pentaho Repository connection. It is required when connecting to the Pentaho Repository
kettle.properties.kettleThis file contains the Kettle engine configuration settings.
installedLicenses.xml.kettle/licensesThis file contains your valid Pentaho license(s). The PDI Enterprise Edition license is required.

Setting up a mounted volume collection

You can declare your own mounted volumes so you can add content to use during execution, such as CSV files. You can also add KTR and KJB files for direct execution.

Create additional Docker volumes

Similar to creating the pdi_kettlevolume, you can create multiple Docker volumes that you can mount inside the container.

Procedure

  1. Run the following command:

    > docker volume create --driver local --opt type=<type-of-volume-?> --opt o=addr=<host-ip>,rw
            --opt device=:<some-dir>/.kettle <docker-volume-name>
  2. Declare any volumes (besides the pdi_kettle volume) that you would like mounted inside of the container as part of the PDI Service/Job volume collection in Foundry.

    The collection of volumes appears under /root/volumes/ inside the container.
  3. (Optional) Because of the way the volume is mounted, you may need to assign write permissions to folders for others (o+w) for any folders serving as the mount point for a volume, specifically in an NFS volume case.

    The docker group is not guaranteed ownership of the folder inside of the container.
    NoteCurrently, PDI Service and PDI Job have different ways of declaring a collection of volumes because of the architectural differences between a Foundry service and Job.

Results

Each volume is mounted as its name inside of this folder in the container.

Example: The sample.csv file is located in your file system at /home/user/sample.csv. You create a Docker volume named csv-samples pointing to /home/user/. This volume is declared in Foundry for a PDI Service/Job.

When properly set up, you would see the sample.csv file at /root/volumes/csv-samples/sample.csv inside of the container.

Declare additional volumes for a PDI Service

Additional volumes can be declared under the Volumes tab of the PDI Service. Any volumes you add must be declared within Docker.

Procedure

  1. From your console, navigate to PDI Service and then select the Volumes tab.

  2. Under PDI_VOLUMES_COLLECTION, edit the volume names and add the Docker volumes you want to mount.

Declare additional mounted volumes for a PDI job

Additional volumes can be declared under the Configuration tab of the PDI Service. Any volumes you add must be declared within Docker.

Procedure

  1. From your console, navigate to PDI Job and then select the Configuration tab.

  2. Navigate to PDI Collection Volume Names.

    This field should contain a comma-separated list of Docker volume names.
  3. Edit the volume names and add the Docker volumes you want to mount.

    CautionDo not add spaces in the comma-separated list of volumes. Otherwise, the volumes and the pdi_kettle volume fail to mount, causing the entire job to fail.