Skip to main content
Hitachi Vantara Lumada and Pentaho Documentation

Executing dataflows

Parent article

This article assumes that you have created a transformation or job in Pentaho Data Integration (PDI) and imported it into Lumada DataOps Suite as a dataflow. See Importing dataflows for instructions.

Execute a dataflow

Perform the following steps to execute a dataflow:

Procedure

  1. From the Card view, locate the dataflow card for the dataflow you want to execute.

  2. Click Execute on the dataflow card.

    The Execution Configuration view displays. You can execute a dataflow from the Execution Configuration page after you enter additional input, or you can set schedules for your dataflow. See Scheduling dataflows for more information.
  3. (Optional) Modify any parameter values you want to use for the execution.

    Depending on your assigned roles and permissions, you can use parameters to customize dataflows. Parameters are created in PDI and are imported with your dataflow. You cannot add or remove dataflow parameters, but you can modify the parameter values. You can also edit how you interact with your parameters. See Edit a dataflow parameter for details.
  4. Click Execute in the upper right corner of the Execution Configuration page to execute the dataflow.

Results

The dataflow executes with the specified parameters.

Next steps

If you need to make small changes to your dataflow, open the Lumada Data Transformation Editor to perform light editing tasks. See Editing dataflows for more information.

Edit a dataflow parameter

When parameters are imported with dataflows into Lumada DataOps Suite, they default to the Text type. Changing the parameter type allows you to execute a dataflow with prompts for other information types. You can edit parameters to change the prompt to one of the following types:
  • Text
  • Checkbox
  • Single selection
  • Multiple selection
  • Date

Perform the following steps to edit the parameters of a dataflow:

Procedure

  1. From the Card view, click the dataflow card you want to modify.

    The Execution Configuration page opens.
  2. (Optional) Click the Visibility icon on the Execution Configuration page if you want to hide the parameter.

    Hiding the parameter prevents a user from changing the value of that parameter. The parameter appears on the Execution report, but does not appear on the Execution Configuration page. The dataflow executes with the default value of the parameter.
  3. Click the Pencil icon next to the parameter that you want to edit.

    The Edit parameter dialog box opens.

    Edit Parameter dialog box

    NoteThe Display name and Prompt type properties marked with an asterisk are required.

Next steps

You can also mask a Text prompt to not display in the Execution report. Single selection and Multiple selection prompt types will display selections or multiple checkboxes, depending on the number of parameter values you assign as shown in the following table:
Number of valuesDisplay
1-3A radio button group
4-9Drop-down menu
10 or moreDrop-down menu with search

Text prompts values can be set as required. Values for Single selection prompts are always required.

View an execution report

Viewing an execution report can help you identify possible issues with a dataflow. To view a report of the latest dataflow execution, click the More actions icon in the bottom-right corner of the dataflow card, then select View execution report.

View execution report option

You can also click the More actions icon on the row for the dataflow on the Monitoring page and click View execution report.

The report is divided into several sections, which are accessible from the left navigation pane:

  • Overview

    A summary that includes the status of the execution, user, and elapsed times for the steps or entries processed.

  • Execution Parameters

    List of execution parameters, if any, and their values.

  • Advanced Settings

    Advanced settings and their values, such as the selected Compute capacity and Log level. To learn more about changing these values, see Advanced settings.

  • Resources

    Shows average, peak, and allocated CPU and Memory resources used by the dataflow. To learn more about changing these values, see Advanced settings.

  • Execution Logs

    Displays line-by-line processing information and total occurrences for each of 3 log levels: Errors, Warnings, and Info. You can filter these results and examine the processed data to determine if issues were encountered. For example, if a step or entry fails, you can view the error with a description of the failure. You can also export a log file for stack trace analysis. See Working with execution logs for more information.

Dataflow execution report

See Monitoring dataflows to view reports of previous executions.

Schedule an execution

You can schedule the execution of a dataflow from the Execution Configuration page.

Learn more

Advanced settings

You can modify the advanced settings for customizations of the compute capacity and the logging level used for dataflow executions. Your system administrator can also set the number of Central Processing Units (CPUs) and memory associated with each compute capacity option.

To access the advanced settings:

  1. From the Flows page, click Execute on the dataflow card. The Execution Settings page displays.
  2. Turn on the toggle under Advanced settings to view the options.
  3. Click the option you want to change.
  4. From the drop-down menu, select the setting you want for the execution or schedule you are creating.

You can also access the advanced settings from the Schedules page by clicking Edit in the More options menu for the schedule you want to modify.

You have control over your computing resources so you can quickly scale capacity as your computing requirements change. The default allocations for the settings are described in the following table.

Compute capacityDescription
Small1 CPU and 2048 MB memory
Medium1.5 CPUs and 3072 MB memory
Large2 CPUs and 4096 MB memory

You can adjust logging levels to capture the correct amount of information needed to analyze the execution of your dataflow. The logging levels include:

Log LevelDescription
NothingNo logging occurs.
ErrorOnly errors are logged.
BasicDefault logging level.
DetailedProvides detailed logging output.
DebugFor debugging purposes, provides very detailed output.
Row LevelLogging at the most detailed level. This log level generates a lot of log data.

All lines in the log are preceded by the date and time of day. Debug and Row Level logging levels contain information which may be too sensitive to be shown to all your users, so you may want to consider the sensitivity of your data when selecting these logging levels. See Monitoring system performance to understand how best to use these logging methods.