The PDI client can pull streaming data from an MQTT broker or clients through an MQTT transformation. The parent MQTT Consumer step runs a child transformation that executes according to the message batch size or duration, allowing you to process a continuous stream of records in near real-time. The child transformation must start with the Get records from stream step.
Additionally, from the MQTT Consumer step, you can select a step in the child transformation to stream records back to the parent transformation. This capability allows records processed by an MQTT Consumer step in a parent transformation to be passed downstream to any other steps included within the same parent transformation.
Select an engine
You can run the MQTT Consumer on the Pentaho engine or on the Spark engine. Depending on your selected engine, the transformation will run differently. Select one of the following options to view how to set up the MQTT Consumer step for your selected engine.
- Using the MQTT Consumer step on the Pentaho engine: Learn how to set up this step when using the Pentaho engine.
- Using the MQTT Consumer step on the Spark engine: Learn how to set up this step when using the Spark engine.
For instructions on selecting an engine from your transformation, see Run configurations.