This article assumes that you have created a transformation or job in Pentaho Data Integration (PDI) and imported it into Lumada DataOps Suite as a dataflow. See Importing dataflows for instructions.
Execute a dataflow
From the Card view, locate the dataflow card for the dataflow you want to execute.
Click Execute on the dataflow card.The Execution Configuration view displays. You can execute a dataflow from the Execution Configuration page after you enter additional input, or you can set schedules for your dataflow. See Scheduling dataflows for more information.
(Optional) Modify any parameter values you want to use for the execution.Depending on your assigned roles and permissions, you can use parameters to customize dataflows. Parameters are created in PDI and are imported with your dataflow. You cannot add or remove dataflow parameters, but you can modify the parameter values. You can also edit how you interact with your parameters. See Edit a dataflow parameter for details.
Click Execute in the upper right corner of the Execution Configuration page to execute the dataflow.
Edit a dataflow parameter
- Single selection
- Multiple selection
Perform the following steps to edit the parameters of a dataflow:
From the Card view, click the dataflow card you want to modify.The Execution Configuration page opens.
(Optional) Click the Visibility icon on the Execution Configuration page if you want to hide the parameter.Hiding the parameter prevents a user from changing the value of that parameter. The parameter appears on the Execution report, but does not appear on the Execution Configuration page. The dataflow executes with the default value of the parameter.
Click the Pencil icon next to the parameter that you want to edit.The Edit parameter dialog box opens.NoteThe Display name and Prompt type properties marked with an asterisk are required.
|Number of values||Display|
|1-3||A radio button group|
|10 or more||Drop-down menu with search|
Text prompts values can be set as required. Values for Single selection prompts are always required.
View an execution report
Viewing an execution report can help you identify possible issues with a dataflow. To view a report of the latest dataflow execution, click the More actions icon in the bottom-right corner of the dataflow card, then select View execution report.
You can also click the More actions icon on the row for the dataflow on the Monitoring page and click View execution report.
The report is divided into several sections, which are accessible from the left navigation pane:
A summary that includes the status of the execution, user, and elapsed times for the steps or entries processed.
List of execution parameters, if any, and their values.
Advanced settings and their values, such as the selected Compute capacity and Log level. To learn more about changing these values, see Advanced settings.
Shows average, peak, and allocated CPU and Memory resources used by the dataflow. To learn more about changing these values, see Advanced settings.
Displays line-by-line processing information and total occurrences for each of 3 log levels: Errors, Warnings, and Info. You can filter these results and examine the processed data to determine if issues were encountered. For example, if a step or entry fails, you can view the error with a description of the failure. You can also export a log file for stack trace analysis. See Working with execution logs for more information.
See Monitoring dataflows to view reports of previous executions.
Schedule an execution
You can schedule the execution of a dataflow from the Execution Configuration page.
You can modify the advanced settings for customizations of the compute capacity and the logging level used for dataflow executions. Your system administrator can also set the number of Central Processing Units (CPUs) and memory associated with each compute capacity option.
To access the advanced settings:
- From the Flows page, click Execute on the dataflow card. The Execution Settings page displays.
- Turn on the toggle under Advanced settings to view the options.
- Click the option you want to change.
- From the drop-down menu, select the setting you want for the execution or schedule you are creating.
You can also access the advanced settings from the Schedules page by clicking Edit in the More options menu for the schedule you want to modify.
You have control over your computing resources so you can quickly scale capacity as your computing requirements change. The default allocations for the settings are described in the following table.
|Small||1 CPU and 2048 MB memory|
|Medium||1.5 CPUs and 3072 MB memory|
|Large||2 CPUs and 4096 MB memory|
You can adjust logging levels to capture the correct amount of information needed to analyze the execution of your dataflow. The logging levels include:
|Nothing||No logging occurs.|
|Error||Only errors are logged.|
|Basic||Default logging level.|
|Detailed||Provides detailed logging output.|
|Debug||For debugging purposes, provides very detailed output.|
|Row Level||Logging at the most detailed level. This log level generates a lot of log data.|
All lines in the log are preceded by the date and time of day. Debug and Row Level logging levels contain information which may be too sensitive to be shown to all your users, so you may want to consider the sensitivity of your data when selecting these logging levels. See Monitoring system performance to understand how best to use these logging methods.