Skip to main content
Hitachi Vantara Lumada and Pentaho Documentation

Monitoring dataflows

Parent article

You can monitor the status and performance of your executed dataflows to verify completed processes and maximize your resource usage. You can view details such as itemized operational metrics, execution duration, and execution status. Additionally, you can view an execution report which provides information about the time, parameters, values, and compute capacity used during processing.

The generated logging data can be sorted by level, searched by keyword, and exported for analysis. To examine the dataflow logs available to your user profile, click Monitoring from the Lumada DataOps Suite menu bar. You can also click the dataflow card to open its Dataflow summary view, where the Monitoring section provides execution information specifically for that dataflow.

Monitor a dataflow

Follow the steps below to monitor the execution of a dataflow.

Procedure

  1. From the Dataflow menu bar, click Monitoring.

    The Monitoring page opens, showing the status of every dataflow execution.Dataflow Monitoring page
  2. Use the Search and Parameters tools to help locate the dataflow that you want to view.

    Use the Search box to filter through dataflow names. Use the Parameters menu to filter through the parameter names and values. See Filter by parameter for details.
  3. Click Preview to see a list of related parameters and their values used during the dataflow execution.

  4. When you locate the dataflow that you want to view, click the More actions icon, then select View execution report.

    The execution report opens. See Working with execution logs for information.

Filter by parameter

Perform the following steps to filter by parameter:

Procedure

  1. Click Parameters in the Monitoring page.

    The Parameters dialog box opens, so you can see the parameters used by dataflow executions in the left pane and possible values of the selected parameters in the right pane.Filter dataflow executions by parameter
  2. Select the parameter and the corresponding parameter values that you want to use in your search for a dataflow execution.

  3. Click Apply.

Results

The filters you applied in the Parameters dialog box appear under Search in the Monitoring page. You can remove a filter by clicking the X to the right of that filter. The table in the Monitoring page updates to include columns of your specified parameters with their values for each dataflow execution found. If you add more parameter columns than can be displayed, you can scroll horizontally to find a specific one.

Working with execution logs

You may want to view the execution log when troubleshooting dataflow execution issues. This log provides you with information and tools for tracking the performance of a dataflow and is part of the execution report. To learn about the sections in the execution report, see View an execution report.

The Execution Log displays at the end of the report. In addition to sorting, filtering, and searching the execution log, you can export the log file to work with later.

The following table describes the execution log features:

IconFeatureDescription
Search iconSearch fieldEnter keywords into this field to search the Execution logs.
N/AAll lines tabClick to show all log lines processed (number of lines in parentheses) during the execution.
Errors iconErrors tab

Click to show only the log lines (number of lines in parentheses) where failures were detected during execution.

Errors must be corrected before the dataflow can be executed successfully. For information about troubleshooting transformations and jobs, see Data integration issues.

Warnings iconWarnings tabClick to show only the log lines (number of lines in parentheses) where warnings were detected during execution. While warnings alert you to potential problems, they do not prevent successful execution.
Info iconInfo tab Click to show only the log lines (number of lines in parentheses) that have processing information.
Download iconDownload tabClick to export the log file. The content is exported in a ZIP file to the browser download directory.
Execution log features

The execution log is comprised of the following columns:

ColumnDescription
Level The level of the log entry: Error, Warning, or Info.
Time offset The time elapsed since the start of dataflow execution.
DateThe date and time the log entry was made.
DescriptionDescription of the log information.
Click the up Up arrow icon and down Down arrow icon arrows in the log table to expand and collapse additional details for each log entry. Log lines that share information gathered from similar processes are grouped together.

Execution log details

If the execution log has many lines, you can browse the log more quickly by using the Execution Duration slider tool. Drag the duration marker to the point in the log you want to view.

Slider appearance during execution

You can even use the Execution Duration slider tool while the dataflow is executing. During execution, the marker displays as white and the Live view notification displays in the upper-right corner. Use the elapsed time next to the Live view notification as a guide to position the duration marker as close as possible to the location or elasped time of the log that you want to view. When finished, click the Live view notification to return to a live view of the execution log.

NoteWhen you navigate using the vertical scrollbar within the log table, the execution duration marker moves to the corresponding location.

When execution is complete, the duration marker is black and the label changes to Latest lines:

Slider appearance after execution

If you use the slider to view the earlier lines in the execution log, a Reset link displays. You can click Reset to return to the end of the log.

Slider appearance while browsing

Export a log

Follow the steps below to export a log file.

Procedure

  1. From the Dataflow menu bar, click Monitoring.

    The Monitoring page opens.
  2. Locate the dataflow that you want to export, click its More actions icon, and then click the View execution report button that appears.

    The execution report page opens.
  3. Click the download icon.

Results

The log is exported as a ZIP file to the download directory of the browser. See Working with log files to review the file contents.

Working with log files

You can use the log file to help identify an issue, and then take corrective action to resolve it or to fine-tune and enhance the performance of the dataflow as needed.

Download the log file as described in Export a log. Then navigate to the file, extract, and open the file to view the contents. The log entry fields in the log files are described in the following table.

FieldDescription
DateTimeStampDate and time of the log.
LogThresholdThe engine’s internal logging level (BASIC, DETAILED, DEBUG, ROWLEVEL, MINIMAL, and ERROR).
MessageLog message.

The following table maps the internal logging levels of the engine to the corresponding logging levels of the execution report.

Engine's internal logging levelDataflow execution report
BASICINFO
DETAILEDINFO
DEBUGINFO
ROWLEVELINFO
MINIMALWARN
ERRORERROR

You can use log entries to understand the individual processes occurring during the execution of your dataflow. Take a look at the following example of a log entry in a log file:

2020-11-16 13:40:23.331 Z [BASIC]: Step Write to log.0 ended successfully, processed 1 lines. ( 0 lines/s)

You can see the time the log entry was made, the threshold of the logging level, the message about the operation being performed, and the results of that process, as mapped in the following table:

FieldDescription
DateTimeStamp2020-11-16 13:40:23.331 Z (where Z is time zone Zulu, UTC+0)
LogThresholdBASIC
MessageStep Write to log.0 ended successfully, processed 1 lines. ( 0 lines/s)