Skip to main content
Version Pentaho
Hitachi Vantara Lumada and Pentaho Documentation

Third-party monitoring with SNMP

  1. Last updated
  2. Save as PDF

Parent article

The main goal of the Simple Network Management Protocol (SNMP) plugin is to provide large enterprises a way to integrate with third-party tools to monitor Data Integration (DI) events. The information received through SNMP will help you figure out the run-time for your transformations or jobs and identify unusually long-running events impacting network performance. These SNMP events can be directly tied to a particular point in time within the execution, such as the moment a transformation started, or a Kettle step finished execution. These events can be used in a broader scope, such as a successful database connection or a Carte server initialization.

The SNMP plugin works by leveraging existing extension points within the Kettle ecosystem to generate events. These events are then converted into SNMP traps, such as unsolicited or asynchronous messages, that are sent along from an SNMP agent. From there, the events are forwarded to the SNMP manager, which is passively listening for such messages.

Parent Topic
Child Topics

Before you begin
To set up a server to send SNMP traps, you will need to download the pentaho-monitoring-plugin-mib-assembly-9.2.0-dist.zip file from the Hitachi Vantara Lumada and Pentaho Support Portal.

Procedure

  1. On the Customer Portal home page, sign in using the Pentaho support user name and password provided in your Pentaho Welcome Packet.

  2. Click Downloads, then click Pentaho 9.2 GA Release in the 9.x list.

  3. On the bottom of the Pentaho 9.2 GA Release page, click the Client Tools > PDI (Spoon) > Plugins folder in the Box widget and download the pentaho-monitoring-plugin-mib-assembly-9.2.0-dist.zip file.

Next steps

To receive trap events, you will need to download and install a third-party SNMP monitoring tool, such as Nagios. Make sure that you also know the IP addresses of your DI and monitoring servers.
Parent Topic

Send traps through a centralized monitoring server

One option for setting up SNMP monitoring on your network is to use one of your existing Pentaho Server as a JMS monitoring server to handle the trap events it receives from a queue. This server then forwards those events available through SNMP to whichever third-party monitoring tool you have installed.

Diagram shows the flow of three JMS events, first to a monitoring server, and then to a third party enterprise monitoring tool
Parent Topic
Child Topics

Step 1: Configure Pentaho Server to send trap events
These steps guide you through the process of configuring the Pentaho Server to work together for SNMP monitoring. The Pentaho Server only needs to know your JMS monitoring server’s IP address to start sending out messages to an event queue. You can also set up the specific types of events you want to monitor. If the servers are on two different machines with different IP addresses, please make sure that they are able to establish communication by pinging.

Procedure

  1. Locate this directory and open the current pentaho-osgi-config-xxx-snmp.cfg file in any text editor.

    pentaho-solutions/system/karaf/system/pentaho/pentaho-osgi-config/xxx

  2. Find the property for fromHost and set the IP address to match your Pentaho Server.

  3. Find the property for toHost and replace it with the IP address of your monitoring server.

  4. If your SNMP port is something other than the default (port 162), change it in this file to match your port.

  5. Configure your other Pentaho Server to NOT perform SNMP monitoring. They are configured to do so by default.

    1. Open the org.apache.karaf.features.cfg file, located in the pentaho-solutions/system/karaf/etc/ directory.

    2. Find the featuresBoot property and verify that pentaho-monitoring-2-snmp and pentaho-monitoring-jms-broker are not in the file. If they are, delete them.

    3. Add pentaho-brokering-jms and pentaho-monitoring-to-snmp within the featuresBoot property.

  6. Save and close the org.apach.karaf.features.cfg file and start the Pentaho Server.

Next steps

Each time you make changes to the pentaho-osgi-config-xxx-snmp.cfg, you need to delete the pentaho.snmp.cfg file located in the pentaho-server/pentaho-solutions/system/karaf/etc/ directory.

A new pentaho.snmp.cfg file is automatically generated based on the updated information contained in pentaho-osgi-config-xxx-snmp.cfg.

After you are done with this procedure, any kettle jobs or transformations that are executed on that Pentaho Server will send SNMP trap events to the designated monitoring server’s IP address.

Parent Topic
  • Send traps through a centralized monitoring server

    One option for setting up SNMP monitoring on your network is to use one of your existing Pentaho Server as a JMS monitoring server to handle the trap events it receives from a queue. This server then forwards those events available through SNMP to whichever third-party monitoring tool you have installed.

Step 2: Configure the monitoring tool to recieve trap events

The Pentaho Server is already configured by default to handle messages. A management information database (MIB) is used for managing entities in a communications network. The structure is hierarchical with each entry assigned an object identifier (OID).

NoteWe are using Nagios as an example in this section, but if you are using a different tool, these processes will be different.
Parent Topic
  • Send traps through a centralized monitoring server

    One option for setting up SNMP monitoring on your network is to use one of your existing Pentaho Server as a JMS monitoring server to handle the trap events it receives from a queue. This server then forwards those events available through SNMP to whichever third-party monitoring tool you have installed.

Child Topics

Upload the MIB file to monitoring tool
These steps will guide you through the process of configuring the server to receive trap events from the Pentaho Server.

Procedure

  1. Access the third-party SNMP tool web interface.

  2. Locate an option for managing MIBs.

  3. Upload the PENTAHO-MIB-V2C.mib file that you downloaded earlier.

Parent Topic
  • Step 2: Configure the monitoring tool to recieve trap events

    The Pentaho Server is already configured by default to handle messages. A management information database (MIB) is used for managing entities in a communications network. The structure is hierarchical with each entry assigned an object identifier (OID).

Convert the MIB file to a configuration file

Procedure

  1. Access the monitoring server through SSH protocol with the root user, for example:

    ssh root@10.100.9.174

  2. After successfully logging in, find the directory containing your uploaded MIB file, such as: /usr/share/snmp/mibs/

  3. Run the snmpttconvertmib command, stating the path to the uploaded file, and the path for the compiled output file:

    snmpttconvertmib
--in=<PATH_TO_UPLOADED_MIB_FILE>
--out=<PATH_TO_COMPILED_OUTPUT_FILE>--
exec='/usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*"'
In this example, we have compiled a configuration file, which we called pentaho.di.conf, based on the previously uploaded Pentaho MIB file:

snmpttconvertmib --in=/usr/share/snmp/mibs/PENTAHO-MIB-V2C.mib --out=/etc/snmp/pentaho.di.conf --exec='/usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*"

Parent Topic
  • Step 2: Configure the monitoring tool to recieve trap events

    The Pentaho Server is already configured by default to handle messages. A management information database (MIB) is used for managing entities in a communications network. The structure is hierarchical with each entry assigned an object identifier (OID).

Add the newly generated configuration file to the SNMP initialization file
Follow these steps to add the .conf file to the snmpt.ini file:

Procedure

  1. Open and edit the file /etc/snmp/snmptt.ini using a text editor.

  2. Locate the [TrapFiles] section; it should be at the bottom of the file.

  3. Place the entry for your newly created config file after the main one but before the END declaration.

    For example, if you use a default snmptt.ini file, it should look something like this:
    [TrapFiles] snmptt_conf_files = <<END
/etc/snmp/snmptt.conf
/etc/snmp/pentaho.di.conf END

  4. Save and close the file.

  5. Restart the monitoring server and the snmpttservice:

    /etc/rc.d/init.d/nagios stop 
/etc/init.d/snmptt stop (wait ~5 seconds)
/etc/init.d/snmptt start
/etc/rc.d/init.d/nagios start
Parent Topic
  • Step 2: Configure the monitoring tool to recieve trap events

    The Pentaho Server is already configured by default to handle messages. A management information database (MIB) is used for managing entities in a communications network. The structure is hierarchical with each entry assigned an object identifier (OID).

Send traps from PDI tools to monitoring tool

Instead of sending traps through a centralized monitoring server first, you can set up your SNMP system to have the trap events sent directly from PDI client tools, such as Pan, Kitchen, or the PDI client. A couple of different ways are available to set up the monitoring application through these tools.

Diagram shows three SNMP events flowing directly into a third party enterprise monitoring tool

Follow one of these options to configure your SNMP environment to run through the PDI client tools.

Parent Topic
Child Topics

From all PDI client tools
Follow this option to configure your SNMP environment to run through the PDI client tools:

Procedure

  1. Locate the below directory and open the org.pentaho.features.cfg file with any text editor. 

    data-integration/system/karaf/system/pentaho/pentaho-osgi-config/6.0.0

  2. Add pentaho-monitoring-to-snmp as a feature at the end of the featuresBoot list.

  3. If your SNMP port is something other than the default (port 162), change it in this file to match your port.

  4. Save and close the file.

Parent Topic
  • Send traps from PDI tools to monitoring tool

    Instead of sending traps through a centralized monitoring server first, you can set up your SNMP system to have the trap events sent directly from PDI client tools, such as Pan, Kitchen, or the PDI client. A couple of different ways are available to set up the monitoring application through these tools.

From a specific PDI client tool such as Carte
Follow this option to configure your SNMP environment to run through a specific PDI client tool, such as Carte:

Procedure

  1. Find the org.pentaho.features.cfg file in that client's etc folder.

  2. Add pentaho-monitoring-to-snmp to the runtimeFeatures list.

    By default, everything BUT the PDI client features are disabled, so you will have to uncomment the runtimeFeatures for the other tools.

  3. If your SNMP port is something other than the default (port 162), you will need to change it in this file to match your port.

  4. Save and close the file.

Parent Topic
  • Send traps from PDI tools to monitoring tool

    Instead of sending traps through a centralized monitoring server first, you can set up your SNMP system to have the trap events sent directly from PDI client tools, such as Pan, Kitchen, or the PDI client. A couple of different ways are available to set up the monitoring application through these tools.

Configuring the type of events to be monitored
The Pentaho Server will monitor by default all possible event types occurring when executing a kettle job or transformation, but you can also specify the type of events you want to monitor by removing items from the list. These steps describe how to specify events.

Procedure

  1. Locate the directory: /pentaho-server/pentaho-solutions/system/karaf/system/pentaho-osgi-config-6.0.0-pdi-monitoring

  2. Open and edit the monitoring.properties file as needed.

  3. Locate the property for extension.point.plugins.enabled, which holds a comma-separated list of event types for which SNMP trap events will be sent to the monitoring server.

    1. Immediately above that property, a comment section lists all possible event types, as well as a brief description for each event type.

    2. Edit that list, leaving only the items you would like to use as SNMP trap events.

  4. Save the file and restart the Pentaho Server.

Parent Topic

Available extension points
Here is a list of the extension points that are used for SNMP monitoring of the Pentaho Server. These events are run by default, but if you want fewer events to run, just remove items from this list and save the file.
Extension PointsDescription
TransformationPrepareExecutionA transformation to prepare execution
TransformationStartA transformation has started
TransformationHeartbeatA signal sent at regular intervals to indicate that the transformation is still active
TransformationFinishA transformation finishes
JobStartA job starts
JobHeartbeatA signal sent at regular intervals to indicate that the job is still active
JobFinishA job finishes
JobBeforeJobEntryExecutionBefore a job entry executes
JobAfterJobEntryExecutionAfter a job entry executes
DatabaseConnectedAfter a successful connection to a database
DatabaseDisconnectedAfter a successful disconnection to a database
CarteStartupRight after the Carte webserver has started
CarteShutdownRight before the Carte webserver shuts down
Parent Topic