Develop your PDI solution

Design helps you think critically about the problem you want to solve and possible solutions. Consider these questions as you gather your requirements and design the solution.

• Output

  What does the overall solution look like? What questions are posing and how do you want the answers formatted?

• Data Sources

  What type(s) of data sources are you querying? Where are they located? How much data do you need to process? Are you using big data? Are you using relational or non-relational data sources? Will you have a target data source? If so, where are they located?

• Content/Processing

  What data quality issues do you have? How is the input data mapped to the output data? Where do you want to process the content, in PDI or in the data source? What hardware will you include in your development environment? Will you need one or more quality assurance test environments or production environments?

Also, consider templates or standards, naming conventions, and other requirements of your end users if you have them. Consider how you will back up your data as well.

Setting up the environment includes installing and configuring PDI on development computers, configuring clustering if needed, and connecting to data sources. If you have one or more quality assurance environments, you will need to set those up also.

Table 1. PDI Set Up Checklist

Task	Do This	Objective
Verify System Requirements	Consult the Components Reference Consult the JDBC Drivers Reference	Acquire one or more servers that meet the requirements. Obtain the correct drivers for your system.
Obtain Software and Install PDI	Install Pentaho PDI Start the Pentaho Server Start Spoon	Get the software from your Sales Support representative. Install the software. Start the Pentaho Server and Spoon.
Install licenses for the Pentaho Server	Install the licenses	Add all relevant Pentaho licenses.
Connect to the Pentaho Repository	Connect to the Pentaho Repository	Connect to the Pentaho Repository.
Apply Advanced Security (if needed)	Review the DI Advanced Security documentation	Determine whether you need to apply DI Advanced Security.

During this step, you develop transformations, jobs, and models, then test what you have developed. You will tune the transformations, jobs, and models for optimal performance.

Development occurs in the Spoon design tool. Spoon’s streamlined design tightly couples the build and test activities so that you can easily perform them iteratively. Spoon has perspectives help you perform ETL and visualize data. Spoon also provides a scheduling perspective that can be used to automate testing. Testing encompasses verifying the quality of transformations and jobs, reviewing visualizations, and debugging issues. One common method of testing is to include steps in a transformation or job that calculate hash totals, checksums, record counts, and so forth to determine whether data is being properly processed. You can also visualize your data in analyzer and report designer and review the results as you develop. This can not only help you find errors and issues with processing, but can help you get a jump on user acceptance testing if you show these reports to your customers or business analysts to get early feedback.

One basic question, is how to determine the numbers of transformations and jobs needed, as well as the order in which they should be executed. A good rule of thumb is to create one transformation for each combination of source system and target tables. You can often identify combinations in your mapping documents. Once you've identified the number of transformations that you need, you can use the same process to determine that number of jobs that you need. When considering the order of execution for transformations and jobs, consider how referential integrity is enforced. Run target table transformations that have no dependencies first, then run transformations that are depend on those tables next, and so forth.

Table 2. Build and Test Checklist - Spoon

Task	Do This	Objective
Understand the Basics	Read Overview of Process	Review information about the process and perspectives.
Review most often used steps and entries	Review commonly-used steps and entries.	Review available transformations and determine how you can use them for your solution. Review job step references to identify which steps can be used in your solution.
Create and Run Transformations	Create and run a transformation.	Identify the transformations needed for your job and implement them. Save transformation. Run transformations locally.
Create and Run a Job	Create and run a job.	Create a job. Arrange transformations in a job so that they execute logically. Run a job.

These resources will be helpful to you as you prepare to Go Live for Production:

• Prepare to Go Live for Production.
• Pentaho Customer Support: check with Support for service packs.