Using Table input to Table output steps with AEL for managed tables in Hive
If you are using managed tables in Hive and want to join a Table input step to a Table output step, use the following workflow when executing on AEL to ensure correct processing. This workflow includes creating separate transformations for the steps and then joining the transformations using a job entry.
See Hive for further configuration information when using Hive with Spark on AEL.
Create separate input and output KTRs
Procedure
Select
A new canvas opens. in the PDI client window to create a new transformation.On the Design tab, click Input and then double-click Table input.
The Table input step appears on the canvas.NoteAs a best practice for smaller managed input tables, use the Copy rows to result step. For larger managed input tables, instead use the Set files in result step.Enter your connection and option information in the Table input step.
Select
then enter a name for the file, such as Table_In. Save the file.Select
A new canvas opens. in the PDI client window to create a new transformation.On the Design tab, click Output and then double-click Table output.
The Table output step appears on the canvas.NoteAs a best practice for smaller managed input tables, use the Get rows from result step. For larger managed input tables, instead use the Get files from result step.Enter your configuration information for the target table in the Table output step.
Click
then enter a name for the file, such as Table_Out. Save the file.
Results
Create a job to join the KTRs
Procedure
Select
A new canvas opens. in the PDI client window to create a new job.On the Design tab, click General and then double-click Start.
The Start entry appears on the canvas.Under General, double-click Transformation.
The Transformation entry appears on the canvas and is connected by a hop from the Start entry.Double-click the Transformation entry.
The Transformation entry dialog box appears.Browse to your saved Table input KTR file, then enter a name, such as Table_In. Click OK to save the entry.
Under General, double-click Transformation.
The Transformation entry appears on the canvas and is connected by a hop from the previous Transformation entry.Double-click the Transformation entry.
The Transformation entry dialog box appears.Browse to your saved Table output KTR file, then enter a name such as Table_Out. Click OK to save the entry.
(Optional) Add a Dummy entry joined by an error hop to each Transformation entry to handle any false results.
Under General, double-click Success.
The Success entry appears on the canvas and is connected by a hop from the Transformation entry.Click
and enter a name for the file. Save it.Press Run to execute the job.
The following example illustrates the job on the canvas:
Results