The logging functionality in Data Integration enables you to more easily troubleshoot complex errors and failures, and measure performance. Use the following steps to turn on logging in Data Integration:
- Create a database or table space called pdi_logging.
- Start Spoon, and open a transformation or job for which you want to enable logging.
- Go to the Edit menu and select Settings... The Settings dialog box appears.
- Select the Logging tab.
- In the list on the left, select the function you want to log.
- Click the New button next to the Log Connection field. The Database Connection dialog box appears.
- Enter your database connection details, then click Test to ensure that they are correct. Click OK when you are done.
- Look through the list of fields to log, and ensure that the correct fields are selected.
Monitoring the LOG_FIELD field can negatively impact Pentaho Server performance. However, if you don't select all fields, including LOG_FIELD, when configuring transformation logging, you will not see information about this transformation in the Operations Mart logging.
Logging is enabled for the job or transformation.
When you run a job or transformation that has logging enabled, you have the following options of log verbosity level in the Run Options window:
|Nothing||Do not record any logging output.|
|Error||Only show errors.|
|Minimal||Only use minimal logging.|
|Basic||This is the default level.|
|Detailed||Give detailed logging output.|
|Debug||For debugging purposes, very detailed output.|
|Row Level||Logging at a row level. This will generate a lot of log data.|
If the Enable time option is selected, all lines in the logging will be preceded by the time of day.
This procedure assumes that you do not have or do not want to use an operating system-level log rotation service. If you are using such a service on your Pentaho server, connect to the Pentaho Server and use that instead of implementing this solution.
The Pentaho server uses the Apache
log4j Java logging framework to store server feedback. The default settings in the
log4j.xml configuration file may be too verbose and grow too large for some production environments. Follow these instructions to modify the settings so that Pentaho server log files are rotated and compressed:
- Stop all relevant servers.
- Download a ZIP archive of the Apache Extras Companion for
log4jpackage: Apache Logging Services.
apache-log4j-extras.jarfile from the ZIP archive, and copy it into
- Edit the
log4j.xmlsettings file for the Pentaho Server. This XML file is located in
- Remove all
PENTAHOCONSOLEappenders from the configuration.
- Modify the
PENTAHOFILEappenders to match the log rotation conditions that you prefer. You may need to consult the
log4jdocumentation to learn more about configuration options. Many Pentaho customers find the following examples useful:
Daily (date-based) log rotation with compression:
<appender name="PENTAHOFILE" class="org.apache.log4j.rolling.RollingFileAppender"> <!-- The active file to log to; this example is for Pentaho Server.--> <param name="File" value="../logs/pentaho.log" /> <param name="Append" value="false" /> <rollingPolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy"> <!-- See javadoc for TimeBasedRollingPolicy --> <param name="FileNamePattern" value="../logs/pentaho.%d.log.gz" /> </rollingPolicy> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d %-5p [%c] %m%n"/> </layout> </appender>
Size-based log rotation with compression:
<appender name="PENTAHOFILE" class="org.apache.log4j.rolling.RollingFileAppender"> <!-- The active file to log to; this example is for Pentaho Server.--> <param name="File" value="../logs/pentaho.log" /> <param name="Append" value="false" /> <rollingPolicy class="org.apache.log4j.rolling.FixedWindowRollingPolicy"> <param name="FileNamePattern" value="../logs/pentaho.%i.log.gz" /> <param name="maxIndex" value="10" /> <param name="minIndex" value="1" /> </rollingPolicy> <triggeringPolicy class="org.apache.log4j.rolling.SizeBasedTriggeringPolicy"> <!-- size in bytes --> <param name="MaxFileSize" value="10000000" /> </triggeringPolicy> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d %-5p [%c] %m%n" /> </layout> </appender>
- Save and close the file, then start all affected servers to test the configuration.
You have an independent log rotation system in place for all modified servers.