Transformation Executor
The Transformation Executor step allows you to execute a Pentaho Data Integration (PDI) transformation. It is similar to the Job Executor step, but works with transformations.
Depending on your data transformation needs, the Transformation Executor step can be set up to function in any of the following ways:
- By default, the specified transformation will be executed once for each input row. You can use the input row to set parameters and variables. The executor step then passes this row to the transformation in the form of a result row.
- You can also pass a group of records based on the value in a field, so that when the field value changes dynamically, the specified transformation is executed. In these cases, the first row in the group of rows is used to set parameters or variables in the transformation.
Note: This function is not currently supported by the Adaptive Execution Layer (AEL). - You can launch multiple copies of this step to assist in parallel transformation processing.
AEL Considerations
The Transformation Executor step can be used to run a sub-transformation with Spark on AEL. When using AEL with the Transformation Executor step, note the following exceptions:
- The Field to use field on the Parameters tab is not supported in AEL.
- The execution results do not show the detailed log of the child transformation in AEL.
Error Handling and Parent Transformation Logging Notes
Keep the following notes in mind when building transformations with the Transformation Executor step:
- This step does not abort when the calling transformation errors out. To control the flow or to abort the transformation in case of errors, please specify the fields and a target step in the Execution results tab to log the number of errors.
- During the actual implementation, the log of the parent transformation contains only the last processed batch of data. This method lessens the strain on the back-end logging. You can obtain a detailed log of the child transformation by viewing the execution results. Be sure to define a target step within the Execution Result tabs and view the field name of the execution logging text, by default, 'ExecutionLogText’.
Samples
These sample transformations demonstrate the capabilities of this step. The samples are available in the distribution package and are located in the design-tools/data-integration/samples/transformations/transformation-executor folder.
-
trans-executor-child.ktr: Adds a sequence to input rows.
-
trans-executor-parent.ktr: Passes rows to a transformation which is then executed three times. You can preview the Results, Result files, and Result rows steps to view the output.
General
Enter the following information in the transformation step fields.
| Options | Description |
|---|---|
| Step Name | Specifies the unique name of the transformation on the canvas. A transformation can be placed on the canvas several times; however, it represents the same transformation. The Step Name is set to 'Transformation Executor' by default. |
| Transformation |
Specify your transformation to execute by entering its path or clicking Browse. If you select a transformation that has the same root path as the current transformation, the variable ${Internal.Entry.Current.Directory} will automatically be inserted in place of the common root path. For example, if the current transformation's path is /home/admin/transformation.ktr and you select a transformation in the folder /home/admin/path/sub.ktr than the path will automatically be converted to ${Internal.Entry.Current.Directory}/path/sub.ktr. If you are working with a repository, specify the name of the transformation. If you are not working with a repository, specify the XML file name of the transformation. Transformations previously specified by reference are automatically converted to be specified by the transformation name within the Pentaho Repository. |
Options
The Transformation Executor step features several tabs with fields for setting parameters and defining results. Each tab is described below.
Parameters Tab
This tab allows you to define or pass variables and/or parameters to the transformation. If multiple rows are passed to the transformation, the first row is used to set the parameters or variables.
| Option | Description |
|---|---|
| Variable/Parameter name | Specify the name of the variable or parameter. |
| Field to use | Specify which field to set a particular parameter or variable value. If you specify an input field to use, the static input value is not used. |
| Static input value | Instead of a field, specify a static value to use. |
| Inherit all variables from transformation (check box) | When this check box is selected, all the variables defined in the current transformation are passed to the specified transformation. |
| Get Parameters (button) | Click this button to insert all the defined parameters of the specified transformation. The description of the parameter is inserted into the static input value field. |
Execution Results Tab
You can define the result fields for the specified transformation and what target step to send them to. If you do not need a certain result, leave a blank input field.
| Option | Description | Default Value |
|---|---|---|
| The target step for the execution results | Use the drop-down menu to select a step in the current transformation as the target step to receive the results from the specified transformation. | N/A |
| Execution time (ms) | Specify the field name for the transformation execution time. | ExecutionTime |
| Execution result | Specify the field name for the transformation execution result. | ExecutionResult |
| Number of errors | Specify the field name for the number of errors during the execution of the transformation. | ExecutionNrErrors |
| Number of rows read | Specify the field name for the total number of rows read during the execution of the transformation. | ExecutionLinesRead |
| Number of rows written | Specify the field name for the total number of rows written during the execution of the transformation. | ExecutionLinesWritten |
| Number of rows input | Specify the field name for the total number of input rows during the execution of the transformation. | ExecutionLinesInput |
| Number of rows output | Specify the field name for the total number of output rows during the execution of the transformation. | ExecutionLinesOutput |
| Number of rows rejected | Specify the field name for the total number of rows rejected during the execution of the transformation. | ExecutionLinesRejected |
| Number of rows updated | Specify the field name for the total number of rows updated during the execution of the transformation. | ExecutionLinesUpdated |
| Number of rows deleted | Specify the field name for the total number of rows deleted during the execution of the transformation. | ExecutionLinesDeleted |
| Number of files retrieved | Specify the field name for the total number of files retrieved during the execution of the transformation. | ExecutionFilesRetrieved |
| Exit status | Specify the field name for the exit status of the execution of the transformation. | ExecutionExitStatus |
| Execution logging text | Specify the field name for the logging text from the execution of the transformation. | ExecutionLogText |
| Log channel ID | Specify the field name for the log channel ID used during the execution of the transformation. | ExecutionLogChannelID |
Row Grouping Tab
Specify how to group the result rows by one of the following methods:
- Specific number of rows.
- Specific field.
- Specified duration of time.
You can use the result rows in a transformation or job entry, or you can get the records themselves by using the Get rows from result step in a transformation.
To access the Field to group rows on or Duration time when collecting rows options, delete the default value in the Number of rows to send to transformation option.
| Option | Group |
|---|---|
| Number of rows to send to transformation | Specify a number. After every 'X' number of rows, the job will be executed and these 'X' rows will be passed to the transformation. |
| Field to group rows on | Specify a field for grouping rows. Rows will be collected in a group as long as the field value stays the same. If the value changes, the transformation will be executed and the accumulated rows will be passed to the transformation. |
| Duration time when collecting rows | Specify a time in milliseconds. This value is the amount of time the step will spend collecting rows prior to the execution of the transformation. |
Result Rows Tab
In this tab, you can specify the destination of the result rows from the transformation execution as well as the layout of the expected result rows.
This step will verify that the data type of the result row fields are identical to what is specified. If there is a difference, an error message will display.
| Option | Description |
|---|---|
| Target step for result rows | Use the drop-down menu to select a step in the current transformation as the target step. |
| Field name | Specify the name of the field. |
| Data type |
Use the drop-down menu to specify the data type of the field, such as number, date, or string. |
| Length | If applicable, specify the length of the data type specified. |
| Precision | If applicable, specify the precision to use. |
Result Files Tab
In this tab, you can specify the destination of the result files.
| Option | Description |
|---|---|
| Target step for result files information | Use the drop-down menu to select a step in the transformation as the target step. |
| Result file name field | Specify the name of the field for the result files. |