Create step plugins

The class implementing StepInterface does not store processing state in any of its fields. Instead an additional class implementing StepDataInterface is used to store processing state, including status flags, indexes, cache tables, database connections, file handles, and alike. Implementations of StepDataInterface declare the fields used during row processing and add accessor functions. In essence the class implementing StepDataInterface is used as a place for field variables during row processing.

PDI creates instances of the class implementing StepDataInterface at the appropriate time and passes it on to the StepInterface object in the appropriate method calls. The base class already implements all necessary interactions with PDI and there is no need to override any base class methods.

A row in PDI is represented by a Java object array, Object[]. Each field value is stored at an index in the row. While the array representation is efficient to pass data around, it is not immediately clear how to determine the field names and types that go with the array. The row array itself does not carry this meta data. Also an object array representing a row usually has empty slots towards its end, so a row can accommodate additional fields efficiently. Consequently, the length of the row array does not equal the amount of fields in the row. The following sections explain how to safely access fields in a row array.

PDI uses internal objects that implement RowMetaInterface to describe and manipulate row structure. Inside processRow() a step can retrieve the structure of incoming rows by calling getInputRowMeta(), which is provided by the BaseStep class. The step clones the RowMetaInterface object and passes it to getFields() of its meta class to reflect any changes in row structure caused by the step itself. Now, the step has RowMetaInterface objects describing both the input and output rows. This illustrates how to use RowMetaInterface objects to inspect row structure.

There is a similar object that holds information about individual row fields. PDI uses internal objects that implement ValueMetaInterface to describe and manipulate field information, such as field name, data type, format mask, and alike.

A step looks for the indexes and types of relevant fields upon first execution of processRow(). These methods of RowMetaInterface are useful to achieve this.

If a step needs to create copies of rows, use the cloneRow() methods of RowMetaInterface to create proper copies. If a step needs to add or remove fields in the row array, use the static helper methods of RowDataUtil. For example, if a step is adding a field to the row, call resizeArray(), to add the field. If the array has enough slots, the orignial array is retruned as is. If the array does not have enough slots, a resized copy of the array is returned. If a step needs to create new rows from scratch, use allocateRowData(), which returns a somewhat over-allocated object array to fit the desired number of fields.

ValueMetaInterface objects are used to determine the characteristics of the row fields. They are typically obtained from a RowMetaInterface object, which is acquired by a call to getInputRowMeta(). The getType() method returns one of the static constants declared by ValueMetaInterface to indicate the PDI field type. Each field type maps to a corresponding native Java type for the actual value. The following table illustrates the mapping of the most frequently used field types.

Do not assume that the Java value of a row field matches these data types directly. This may or may not be true, based on the storage type used for the field.

In addition to the data type of a field, the storage type, getStorageType()/setStorageType(), is used to interpret the actual field value in a row array. These storage types are available.

In a typical data processing scenario, a step is not interested in dealing with the complexities of the storage type. It just needs the actual data value on which to do processing. In order to safely read the value of a field, the ValueMetaInterface object provides a set of accessor methods to get at the actual Java value. The argument is a value from a row array that corresponds to the ValueMetaInterface object. The accessor methods always return a proper data value, regardless of the field storage type.

• getString()
• getInteger()
• getNumber()
• getBigNumber()
• getDate()
• getBoolean()
• getBinary()

For each of these methods, RowMetaInterface has corresponding methods that require the row array and the index of the field as arguments.

ValueMetaInterface represents all aspects of a PDI field, including conversion masks, trim type, and alike. All of these are available using corresponding accessor methods, such as getConversionMask() and getTrimType(). Refer to the Javadoc for a complete overview.

Transformation steps may encounter errors at many levels. They may encounter unexpected data, or problems with the execution environment. Depending on the nature of the error, the step may decide to stop the transformation by throwing an exception, or support the PDI error handling feature, which allows you to divert bad rows to an error handling step.

If a step encounters an error during row processing, it may log an error and stop the transformation. This is done by calling setErrors(1), stopAll(), setOutputDone(), and returning false from processRow(). Alternatively, the step can throw a KettleException, which also causes the transformation to stop.

It is sensible to stop the transformation when there is a problem with the environment or configuration of a step. For example, when a database connection cannot be made, a required file is not present, or an expected field is not in the row stream. These are errors that affect the execution of the transformation as a whole. If on the other hand the error is related to row data, the step should implement support for the PDI Error Handling feature.

You may want to divert bad rows to a specific error handling step. This capability is referred to as the Error Handling feature. A step supporting this feature overrides the BaseStep implementation of supportsErrorHandling() to return true. This enables you to specify a target step for bad rows in the PDI client UI. During runtime, the step checks if you configured a target step for error rows by calling getStepMeta().isDoingErrorHandling(). If error rows are diverted, the step passes the offending input row to putError() and provides additional information about the errors encountered. It does not throw a KettleException. If you do not configure a step to generate error rows and send them to another step for processing, the step falls back to calling a hard stop.

Most core PDI steps support row-level error handling. The Number Range step is a good example. If error handling is enabled, it diverts the row into the error stream. If it is not, the step stops the transformation.

During transformation execution, each PDI step keeps track of a set of step metrics. These are displayed in the PDI client (Spoon) in the Step Metrics tab.

Each step metric is essentially a row counter. The counters are manipulated by calling the corresponding increment, decrement, get, and set methods on BaseStep. This table provides a list of the counters and the correct way to use them.

A step interacts with the PDI logging system by using the logging methods inherited from BaseStep.

These methods are used to issue log lines to the PDI logging system on different severity levels. Multi-argument versions of the methods are available to do some basic formatting, which is equivalent to a call to MessageFormat.format(message, arguments).

• public void logMinimal()
• public void logBasic()
• public void logDetailed()
• public void logDebug()
• public void logRowlevel()
• public void logError()

These methods query the logging level. They are often used to guard sections of code, that should only be executed with elevated logging settings.

• public boolean isBasic()
• public boolean isDetailed()
• public boolean isDebug()
• public boolean isRowLevel()

Steps should log this information at specified levels.

A transformation defines a feedback size in its settings. The feedback size defines the number of rows after which each step logs a line reporting its progress. This is implemented by calling checkFeedback() with an appropriate row counter as argument to determine if feedback should be logged. Feedback logging happens on the basic log-level. There are many ways you can implement this. Here is an example implementation snippet.

if (checkFeedback(getLinesWritten())) {
  if(isBasic()) logBasic("linenr "+getLinesWritten());
} 

It may make sense to use different row counters for checking the feedback size depending on the implementation logic of your step. For example, a step that accumulates incoming rows into one single summary row, should probably use the linesRead counter to determine the feedback interval.

The Excel Output step has a good example implementation of feedback logging.

Use the @InjectionSupported annotation in your step meta class to indicate that it supports metadata injection. This annotation has the following parameters.

For example, setting the localizationPrefix parameter to “Injection.” for the “FILENAME” field indicates the <step_package>/messages/messages_Injection.FILENAME.properties file. This prefix and "FILENAME" field within the following @InjectionSupported annotation tells the system to use the key "Injection.FILENAME" to retrieve descriptions along with the optional “GROUP1” and “GROUP2" groups.

@InjectionSupported(localizationPrefix="Injection.", groups = {"GROUP1","GROUP2"})

If your step already has metadata injection support using a pre-6.0 method, such as it returns an object from the getStepMetaInjectionInterface() method, then you will need to remove the injection class and getStepMetaInjectionInterface() method from the step meta class. After this class and method are removed, the method getStepMetaInjectionInterface() is called from the base class (BaseStepMeta), and returns null. The null value indicates your step does not support pre-6.0 style metadata injection. Otherwise, if your step did not use this type of implementation, you do not need to add or manually modify this method to the step meta class.

Although inheritance applies to injectable fields specified by the @Injection and @InjectionDeep annotations, you still need to apply the @InjectionSupported annotation to any step inheriting the injectable fields from another step. For example, if an existing input step has already specified injectable fields through the @Injection annotation, you do not need to use the @Injection annotation for fields you inherited within the step you create. However, you still need to use the @InjectionSupported annotation in your step meta class even though that annotation is also already applied in the existing input step.

Each field (or setter) you want to be injected into your step should be marked by the @Injection annotation. The parameters of this annotation are the name of the injectable field and the group containing this field:

@Injection(name = "FILENAME", group = "FILE_GROUP") - on the field or setter

This annotation has the following parameters.

This annotation can be used either:

• On a field with simple type (string, integer, float, etc.)
• On the setter of a simple type
• In an array of simple types
• On a java.util.List of simple types. For this List usage, type should be declared as generic.

Besides with these types, you need to understand special exceptions for enums, arrays, and data type conversions.

You can mark any enum field with the @Injection annotation. For enum fields, metadata injection converts source TYPE_STRING values into enum values of the same name. For your user to be able to use any specified values, all possible values should be described in the documentation of your metadata injection step.

Any @Injection annotation can be added to an array field:

@Injection(name="FILES")
private String[] files;

The metadata object can also have a more complex structure:

MyStepMeta.java

public class MyStepMeta {

  @InjectionDeep
  private OneFile[] files;


  public class OneFile {
    @Injection(name="NAME", group="FILES")
    public String name;
    @Injection(name="SIZE", group="FILES")
    public int size;
  }
}

Metadata injection creates objects for each row from the injection information stream. The number of objects equals the number of rows in the information stream. If different injections (like NAME and SIZE in the example above) are loaded from different information streams, you have to make sure that the row numbers are equal on both streams.

You can convert from RowSet to simple type for a field with the DefaultInjectionTypeConverter class. Currently supported data types for fields are string, boolean, integer, long, and enum. You can also define non-standard custom converters for some fields by declaring them in the 'converter' attribute of @Injection annotation. These custom conversations are extended from the InjectionTypeConverter class.

Only the fields of the step meta class (and its ancestors) are checked for annotations, which works well for simple structures. If your step meta class contains more complex structures (beyond primitive types), you can use the @InjectionDeep annotation with an optional prefix parameter to inspect annotations inside these complex (not primitive) fields:

@InjectionDeep(prefix = "MyPrefix")

This annotation can be used on the array or java.util.List of complex fields.

The optional prefix parameter enables you to use a class in the @InjectionDeep annotation more than once and still keep unique metadata injection property names. For example, MyStepMeta has more than one Info field. By specifying "FIRST" and "LAST" prefixes through the @InjectionDeep annotation, the metadata injection property structures of "FIRST.NAME" and "LAST.NAME" can be uniquely identified:

public class Info {
  @Injection(name = "NAME")
  String name;
}

public class MyStepMeta {
  @InjectionDeep(prefix = "FIRST")
  Info first;

  @InjectionDeep(prefix = "LAST")
  Info last;
}

You can use generic object types in your declared variables, arrays, or lists. Your step meta class must extend or implement a base which uses generics. For example, if you have a base class named MyBaseMeta, which makes use of generic types, you need to extend it within the step meta class:

@InjectionSupported(...)
  public class MyMeta extends MyBaseMeta<String> {
}

Your base class (MyBaseMeta) could then be defined with any of the following generics:

public abstract class MyBaseMeta<T> {
   @Injection(...)
   T name;

   @Injection(...)
   T[] children;

   @Injection(...)
   List<T> additional;
}

ou can also use generics with the @InjectionDeep annotation. For example:

public abstract class MyBaseMeta<T> {
  @InjectionDeep
  T info;

  @InjectionDeep
  T[] children;

  @InjectionDeep
  List<T> additional;
}