Transformation Step Reference

Add sequence depending of fields value change. Each time value of at least one field change, PDI will reset sequence.

Generate documentation automatically based on input in the form of a list of transformations and jobs.

Decode binary or JSON Avro data and extracts fields from the structure it defines, either from flat files or incoming fields.

Block flow until all incoming rows have been processed. Subsequent steps only recieve the last input row to this step.

Update a junk dimension in a data warehouse. Alternatively, look up information in this dimension. The primary key of a junk dimension are all the fields.

Concatenate multiple fields into one target field. The fields can be separated by a separator and the enclosure logic is completely compatible with the Text File Output step.

Write rows to the executing job. The information will then be passed to the next entry in this job.

Retrieve all documents from a given view in a given design document from a given database.

Determines: (1) if a credit card number is valid (uses LUHN10 (MOD-10) algorithm) (2) which credit card vendor handles that number (VISA, MasterCard, Diners Club, EnRoute, American Express (AMEX),...).

Output one empty row if input stream is empty (ie when input stream does not contain any row).

Update a slowly changing dimension in a data warehouse. Alternatively, look up information in this dimension.

Does not do anything. It is useful however when testing things or in certain situations where you want to split streams.

Execute dynamic SQL statement build in a previous field.

Convert an Edifact message to XML to simplify data extraction (Available in PDI 4.4, already present in CI trunk builds).

Read POP3/IMAP server and retrieve messages.

Read shape file data from an ESRI shape file and linked DBF file.

Is an example of a plugin test step.

Execute a process and return the result.

Find the approximate matches to a string using matching algorithms. Read a field from a main stream and output approximative value from lookup stream.

Generate random valide (luhn check) credit card numbers.

Generate random value.

Generate a number of empty or equal rows.

Get data from XML file by using XPath. This step also allows you to parse XML defined in a previous field.

Get file names from the operating system and send them to the next step.

Read filenames used or generated in a previous entry in a job.

Get files rows count.

Retrieve unique IDs in blocks from a slave server. The referenced sequence needs to be configured on the slave server in the XML configuration file.

Return records that were previously generated by another transformation in a job. 

List detailed information about transformations and/or jobs in a repository.

Read rows from a previous entry in a job.

Retrieve the value of a session variable.

Read a parent folder and return all subfolders.

Get information from the system like system date, arguments, etc.

Get table names from database connection and send them to the next step.

Determine the values of certain (environment or Kettle) variables and put them in field values.

Fetch data from google analytics account.

Bulk load Greenplum data. Replacement step is Greenplum Load.

Bulk load Greenplum data. 

Build aggregates in a group by fashion. This works only on a sorted input. If the input is not sorted, only double consecutive rows are handled correctly.

Read in parallel from a GZIP CSV file.

Read data from a variety of different text-file types stored on a Hadoop cluster.

Write data to a variety of different text-file types stored on a Hadoop cluster.

Read from an HBase column family.

Write to an HBase column family.

Decode an incoming key and HBase result object according to a mapping.

Read data from HL7 data streams.

Call a web service over HTTP by supplying a base URL by allowing parameters to be set dynamically.

Call a web service request over HTTP by supplying a base URL by allowing parameters to be set dynamically.

Receive messages from any IBM Websphere MQ Server. 

Send messages to any IBM Websphere MQ Server. 

Mark the last row.

Set a field value to a constant if it is null.

Load data to an Infobright database table.

Interface with the Ingres VectorWise Bulk Loader "COPY TABLE" command.

Inject rows into the transformation through the java API.

Update or insert rows in a database based upon keys.

Filter rows using java code.

Receive messages from a JMS server.

Send messages to a JMS server.

Run a PDI job, and passes parameters and rows. 

Output the cartesian product of the input streams. The number of rows is the multiplication of the number of rows in the input streams.

Extract relevant portions out of JSON structures (file or incoming field) and output rows.

Create JSON block and output it in a field to a file.

Run a sub-transformation that executes according to message batch size or duration, letting you process a continuous stream of records in near-real-time.

Publish messages in near-real-time across worker nodes where multiple, subscribed members have access.

Executes a Knowledge Flow data mining process.

Read data from LDAP host.

Perform Insert, upsert, update, add or delete operations on records based on their DN (Distinguished Name).

Read data from LDIF files.

Load file content in memory.

Load data into LucidDB by using Remote Rows UDX. 

Send e-mail.

Check if an email address is valid.

Run a mapping (sub-transformation), use MappingInput and MappingOutput to specify the fields interface.

Specify the input interface of a mapping.

Specify the output interface of a mapping.

Enter Key Value pairs from Hadoop MapReduce.

Exit Key Value pairs, then push into Hadoop MapReduce.

Lookup an IPv4 address in a MaxMind database and add fields such as geography, ISP, or organization.

Build aggregates in a group by fashion. This step doesn't require sorted input.

Join two streams on a given key and outputs a joined set. The input streams must be sorted on the join key.

Merge two streams of rows, sorted on a certain key. The two streams are compared and the equals, changed, deleted and new rows are flagged.

Read the metadata of the incoming stream.

Read data from a Microsoft Access file

Store records into an MS-Access database table.

Read data from Excel and OpenOffice Workbooks (XLS, XLSX, ODS).

Store records into an Excel (XLS) document with formatting information.

Write or appends data to an Excel file.

Run JavaScript programs (and much more).

Execute and retrieve data using an MDX query against a Pentaho Analyses OLAP server (Mondrian).

Load data into MonetDB by using their bulk load command in streaming mode.

Read all entries from a MongoDB collection in the specified database.

Write to a MongoDB collection.

Join multiple streams. This step supports INNER and FULL OUTER joins.

Load data over a named pipe (not available on MS Windows).

Set a field value to null if it is equal to a constant value.

Create ranges based on numeric field.

Execute and retrieve data using an MDX query against any XML/A OLAP datasource using olap4j.

Delete data from the OpenERP server using the XMLRPC interface with the 'unlink' function. 

Retrieve data from the OpenERP server using the XMLRPC interface with the 'read' function.

Update data on the OpenERP server using the XMLRPC interface and the 'import' function 

Use Oracle Bulk Loader to load data.

Return metrics for one or several steps.

Retrieve all cell data from a Palo cube.

Update cell data in a Palo cube.

Return elements from a dimension in a Palo database.

Create/update dimension elements and element consolidations in a Palo database.

Decode Parquet data formats and extracts fields from the structure it defines.

Map fields within data files and choose where you want to process those files.

Execute an existing report file (.prpt).

Bulk load PostgreSQL data.

Prioritize streams in an order way.

Process one file per row (copy or move or delete). This step only accept filename in input.

Write data to properties file.

Read data (key, value) from properties files.

Execute an R script within a PDI transformation.

Evaluate regular expressions. This step uses a regular expression to evaluate a field. It can also extract new fields out of an existing field with capturing groups.

Replace all occurrences a word in a string with another word.

Transform Samples a fixed number of rows from the incoming stream.

Consume RESTful services. REpresentational State Transfer (REST) is a key design idiom that embraces a stateless client-server architecture in which the web services are viewed as resources and can be identified by their URLs

Denormalise rows by looking up key-value pairs and by assigning them to new fields in the output rows. This method aggregates and needs the input rows to be sorted on the grouping fields.

Flatten consecutive rows based on the order in which they appear in the input stream.

Normalise de-normalised information.

Read RSS feeds.

Read RSS stream.

Execute a rule against each row (using Drools).

Execute a rule against a set of rows (using Drools).

Run SSH commands and returns result.

Turn a repetitive, re-usable part of a transformation (a sequence of steps) into a mapping (sub-transformation).

Execute a sequence of steps in a single thread. 

Compare the data from two tables (provided they have the same lay-out). It'll find differences between the data in the two tables and log it.

Read data from a text file in several formats. This data can then be passed to your next step(s).

Write rows to a text file.

Run a Pentaho Data Integration transformation, sets parameters, and passes rows.

Remove double rows and leave only unique occurrences. This works only on a sorted input. If the input is not sorted, only double consecutive rows are handled correctly.

Remove double rows and leave only unique occurrences by using a HashSet.

Read YAML source (file or stream) parse them and convert them to rows and writes these to one or more output.