Enable Logging

Last updated
Save as PDF

The logging functionality in Data Integration enables you to more easily troubleshoot complex errors and failures, and measure performance. Use the following steps to turn on logging in Data Integration:

Create a database or table space called pdi_logging.
Start Spoon, and open a transformation or job for which you want to enable logging.
Go to the Edit menu and select Settings... The Settings dialog box appears.
Select the Logging tab.
In the list on the left, select the function you want to log.
Click the New button next to the Log Connection field. The Database Connection dialog box appears.
Enter your database connection details, then click Test to ensure that they are correct. Click OK when you are done.
Look through the list of fields to log, and ensure that the correct fields are selected.

Monitoring the LOG_FIELD field can negatively impact Pentaho Server performance. However, if you don't select all fields, including LOG_FIELD, when configuring transformation logging, you will not see information about this transformation in the Operations Mart logging.

Logging is enabled for the job or transformation.

When you run a job or transformation that has logging enabled, you have the following options of log verbosity level in the Run Options window:

Log Level	Description
Nothing	Do not record any logging output.
Error	Only show errors.
Minimal	Only use minimal logging.
Basic	This is the default level.
Detailed	Give detailed logging output.
Debug	For debugging purposes, very detailed output.
Row Level	Logging at a row level. This will generate a lot of log data.

If the Enable time option is selected, all lines in the logging will be preceded by the time of day.

Log Rotation

This procedure assumes that you do not have or do not want to use an operating system-level log rotation service. If you are using such a service on your Pentaho server, connect to the Pentaho Server and use that instead of implementing this solution.

The Pentaho server uses the Apache log4j Java logging framework to store server feedback. The default settings in the log4j.xml configuration file may be too verbose and grow too large for some production environments. Follow these instructions to modify the settings so that Pentaho server log files are rotated and compressed:

Stop all relevant servers.
Download a ZIP archive of the Apache Extras Companion for log4j package: Apache Logging Services.
Unpack the apache-log4j-extras.jar file from the ZIP archive, and copy it into server/pentaho-server/tomcat/webapps/pentaho/WEB-INF/lib/.
Edit the log4j.xml settings file for the Pentaho Server. This XML file is located in server/pentaho-server/tomcat/webapps/pentaho/WEB-INF/classes/.
Remove all PENTAHOCONSOLE appenders from the configuration.

Modify the PENTAHOFILE appenders to match the log rotation conditions that you prefer. You may need to consult the log4j documentation to learn more about configuration options. Many Pentaho customers find the following examples useful:

Daily (date-based) log rotation with compression:

<appender name="PENTAHOFILE" class="org.apache.log4j.rolling.RollingFileAppender">
    <!-- The active file to log to; this example is for Pentaho Server.-->
    <param name="File" value="../logs/pentaho.log" />
    <param name="Append" value="false" />
    <rollingPolicy class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
        <!-- See javadoc for TimeBasedRollingPolicy -->
        <param name="FileNamePattern" value="../logs/pentaho.%d.log.gz" />
    </rollingPolicy>
    <layout class="org.apache.log4j.PatternLayout">
        <param name="ConversionPattern" value="%d %-5p [%c] %m%n"/>
    </layout>
</appender>

Size-based log rotation with compression:

<appender name="PENTAHOFILE" class="org.apache.log4j.rolling.RollingFileAppender">
    <!-- The active file to log to; this example is for Pentaho Server.-->
    <param name="File" value="../logs/pentaho.log" />
    <param name="Append" value="false" />
    <rollingPolicy class="org.apache.log4j.rolling.FixedWindowRollingPolicy">
        <param name="FileNamePattern" value="../logs/pentaho.%i.log.gz" />
        <param name="maxIndex" value="10" />
        <param name="minIndex" value="1" />
    </rollingPolicy>
    <triggeringPolicy class="org.apache.log4j.rolling.SizeBasedTriggeringPolicy">
        <!-- size in bytes -->
        <param name="MaxFileSize" value="10000000" />
    </triggeringPolicy>
    <layout class="org.apache.log4j.PatternLayout">
        <param name="ConversionPattern" value="%d %-5p [%c] %m%n" />
    </layout>
</appender>

Save and close the file, then start all affected servers to test the configuration.

You have an independent log rotation system in place for all modified servers.