Skip to main content
Hitachi Vantara Lumada and Pentaho Documentation

Manage data routes

Control the flow of data from a device to a protocol or database.

You can create and manage a data route for a data source to one or more valid destinations. Destinations can be either protocols or databases.

A diagram of a data route, from source, to broker, to routing services, and then to a data destination either directly or through a broker and application.

Data route configuration values

The following tables describe the protocol and database options supported in IIoT Core Services.

Data sources

Gateways can receive data using the MQTT or Modbus protocol. IIoT Core can receive data through AMQP or REST.

AMQP protocol properties for Core
Property Description AMQP - Lumada Default Setting
Queue Name (Data Source only) The queue name to publish the message. Each asset must have a unique Queue Name.Specify the asset name appended with a serialized number and the word ‘queue.’
Hostname / IP Address The name of the host. IP-address
Port The port number to connect to the AMQP broker. 30671
Virtual HostThe virtual host path, if needed. Example: entering "mypath" becomes "<host>:<port>/mypath". This property is not displayed if AMQP - Lumada Default Setting is selected as the Protocol.N/A
Username The user name used for authentication with the AMQP broker. admin
Password The password used for authentication with the AMQP broker. This property is not displayed if AMQP - Lumada Default Setting is selected as the protocol.(Ask your administrator)
Binding Key The binding or routing key is a message attribute.

The data source Binding Key must match the Trace ID if the value for the Data Type parameter is application/json. If the Data Type value is application/x-hiota, the Trace ID must match the Trace ID in the Common Data Model. See Common Data Model.

Depending on the exchange type, the exchange might use this key to determine how to route messages to queues.

For Data Source, the value defaults to asset-name with the extension "route". For Data Destination, it defaults to asset-name-destination.

Quality of Service(Data Source only) The maximum amount of unacknowledged deliveries that are permitted on a channel. When the maximum number is reached, no more messages are delivered until at least one message is acknowledged. 10
Exchange Name Topic to publish or subscribe to depending on the direction of the route. hiota-exchange (not editable)
Exchange Type The functionality of the exchange, for example, how messages are routed through it. Valid values:
  • direct
  • fanout
  • topic
topic (not editable)
Reconnect Interval (Seconds)The amount of time, in seconds, between reconnection attempts. Default value is 5 seconds. This property is not displayed if AMQP - Lumada Default Setting is selected as the protocol.N/A
Allow Untrusted CA CertificatesSelf-signed certificates. Enable (default) to allow, disable to allow only trusted certificates. This property is not displayed if AMQP - Lumada Default Setting is selected as the protocol.N/A
Kafka protocol properties for Core
Property Description Kafka - Lumada Default Setting
Hostname / IP Address The name of the host. IP-address
Port The port number to connect to the Kafka broker. 30092
Username The user name used for authentication with the Kafka broker. N/A
Password The password used for authentication with the Kafka broker. This property is not displayed if Kafka - Lumada Default Setting is selected as the protocol.N/A
Topic Category or feed name to which records are published. Kafka messages are organized in topics.N/A
TimeoutDuration of the connection to Kafka.2s
MQTT protocol properties for gateway
PropertyDescriptionLumada Default Setting
TopicThe topic to publish or subscribe to. asset topic
Hostname / IP AddressThe host of the gateway device you want to collect data from.gateway host
PortThe port for the gateway.gateway port
UsernameThe user name required to log in to the gateway.N/A
PasswordThe password required to log in to the gateway.N/A
Quality of ServiceThe MQTT quality of service to use for message exchanges. Values are AtMostOnce (0), AtLeastOnce (1), or ExactlyOnce (2).ExactlyOnce (2)
QuiesceThe maximum number of seconds to wait for the gateway to pause before disconnecting.10 seconds
Dispatch methodMQTT or REST, for sending data from an IIoT Gateway to IIoT Core. REST is selected automatically when the Data Size is set to large. Store and forward is supported by default for MQTT.MQTT
Modbus protocol properties for gateway
Property Description Modbus - Lumada Default Setting
Hostname / IP Address The Modbus server host IP address. N/A
Host Port The server port number. The default port number is 502.N/A
Protocol The interface that Modbus supports.TCP
Operation Read operation. Read
Cardinality The start of the register or coil. N/A
Start Address The start address of the register or coil. N/A
Slave Address

Addresses of the devices supplying information.

Supports up to 247 secondary addresses.
N/A
Poll Interval Number of seconds between polls. 10 seconds
Data Descriptors The tags sent from Modbus, for example, Register [50] = Speed. N/A

Data destinations

The following tables describe the options for the database or protocol that you select as the Destination Type.

CouchDB database properties
Property Description Lumada Default Setting
Hostname / IP Address The name of the host on which the database resides. IP-address
Port The port number for the host. 30984, 30224 (Passport API)
Username The user name for the database connection. admin
Password The password for the user name. This property is not displayed if COUCH - Lumada Default Setting is selected. (Ask your administrator)
Database NameThe asset selected during route creation. Each asset must have a unique queue name.asset name with the extension "db"
Allow Untrusted CA CertificatesSelf-signed certificates. Select to allow, deselect to allow only trusted certificates. This property is not displayed if COUCH - Lumada Default Setting is selected.N/A
Save IT Data (Optional) This causes fields to be saved in the IT Data section of the CDM. Disabled
Enable Store and ForwardWhen enabled, messages are written to disk when the connection is disrupted, and forwarded when the connection is restored.Yes
Store and Forward Number of RetriesNumber of times to attempt forwarding.10
Store and Forward Notification IntervalInterval between forwarding attempts. Enter a number followed by ‘s’ for seconds, ‘h’ for hours, or ‘d’ for days. 1s
InfluxDB database properties
Property Description Lumada Default Setting
Hostname / IP Address The name of the host on which the database resides. IP-address
Port The port number for the host. 30086, 30223 (Passport API)
OrganizationA way for InfluxDB to organize data.N/A
TokenSecurity token needed to access the InfluxDB database.N/A
BucketName of the InfluxDB database section used.<asset-name>_db_<number>
Retention PolicyThe length of time to retain messages. Enter a number followed by ‘s’ for seconds, ‘h’ for hours, or ‘d’ for days. No data is purged if the value is left blank.N/A
Timeout The number of seconds requests wait for the client to complete a database transaction. 1s
PrecisionHow frequently the InfluxDB database saves the data.nanoseconds
Save IT Data (Optional) This causes fields in the IT Data section of the CDM to be saved. Disabled
Allow Untrusted CA CertificatesSelf-signed certificates. Select to allow, deselect to allow only trusted certificates. This property is not displayed if the INFLUX - Lumada Default Setting is selected.Enabled
Enable Store and ForwardWhen enabled, messages are written to disk when the connection is disrupted, and forwarded when the connection is restored.Enabled
Store and Forward number of retriesNumber of times to attempt forwarding.10
Store and Forward Notification IntervalInterval between forwarding attempts. Enter a number followed by ‘s’ for seconds, ‘h’ for hours, or ‘d’ for days. 1s
MinIO database properties
Property Description Lumada Default Setting
Hostname / IP Address The name of the host on which the database resides. IP-address
Port The port number for the host. 31000
Access Key The MinIO access key. Automatically generated or defined during installation.
Secret Key The MinIO secret key. This property is not displayed if MINIO - Lumada Default Setting is selected. (Ask your administrator)
Bucket The asset selected during route creation. This property can only contain lowercase alphanumeric characters or dashes and must start with a letter.asset name
Timeout The timeout duration in seconds when trying to connect to the database.10
Idle Timeout The timeout duration for requests, in seconds.10
TLS Timeout The timeout duration in seconds for TLS handshakes. 10
Expect Continue Timeout The expect continue timeout, in seconds. 10
Keep Alive The keep alive time for requests, in seconds. 10
Max Idle Connections The maximum number of idle connections that can occur. 10
Allow Untrusted CA CertificatesSelf-signed certificates. Enable (default) to allow, disable to allow only trusted certificates. This property is not displayed if MINIO - Lumada Default Setting is selected.N/A
Compression Database compression. Enabled
Enable Store and ForwardWhen enabled, messages are written to disk when the connection is disrupted, and forwarded when the connection is restored.Yes
Store and Forward Number of RetriesNumber of times to attempt forwarding.10
Store and Forward Notification IntervalInterval between forwarding attempts. Enter a number followed by 's' for seconds, 'h' for hours, or 'd' for days.1s
MongoDB database properties
Property Description Lumada Default Setting
Database NameThe asset selected during route creation. Each asset must have a unique queue name.asset name with the extension "db"
CollectionDatabase collection name to store the data.asset name with the extension "table"
Save in Hiota(Optional) This causes fields in the Hiota database to be saved.Disabled
Enable Store and ForwardWhen enabled, messages are written to disk when the connection is disrupted and forwarded when the connection is restored.Yes
Store and Forward Number of RetriesNumber of times to attempt forwarding.10
Store and Forward Notification IntervalInterval between forwarding attempts. Enter a number followed by ‘s’ for seconds, ‘h’ for hours, or ‘d’ for days.1s
Postgres database properties
Property Description
ColumnName of the column. Each column has a unique name.
TypeData types for the column:
  • smallint
  • int
  • bigint
  • bool
  • varchar
  • real
  • float
  • timestamptz
  • bytea
  • text
  • json
  • uuid
SizeMaximum data size for the column. Only applies for varchar and binary.
ConstraintsThe following constraints are supported:
  • primary Key
  • not null
  • unique
  • default (n)
Hostname / IP AddressThe host IP address where Postgres resides.
PortPort number of the host.
UsernameUser name to log in to the host.
PasswordPassword to log in to the host.
DatabaseDatabase name.
TableTable name in the database to store the data.
Store and Forward number of retriesNumber of times to attempt forwarding.
Store and Forward notification intervalInterval between forwarding attempts.
AMQP protocol properties for Core
Property Description Lumada Default Setting
Hostname / IP AddressThe name of the host. IP-address
Port The port number to connect to the AMQP broker. 30671
Virtual Host

The virtual host path, if needed.

Example: Entering "mypath" generates "<host>:<port>/mypath".

This property is not displayed if AMQP - Lumada Default Setting is selected as the Protocol.

N/A
Username The user name used for authentication with the AMQP broker. admin
Password The password used for authentication with the AMQP broker. This property is not displayed if AMQP - Lumada Default Setting is selected as the protocol.(Ask your administrator)
Binding Key The binding or routing key is a message attribute.

The data source Binding Key must match the Trace ID if the value for the Data Type parameter is application/json. If the Data Type value is application/x-hiota, the Trace ID must match the Trace ID in the Common Data Model. See Common Data Model.

Depending on the exchange type, the exchange might use this key to determine how to route messages to queues.

For Data Source, the value defaults to asset name with the extension "route". For Data Destination, it defaults to asset name destination.

Exchange Name Topic to publish or subscribe to depending on the direction of the route. hiota-exchange (not editable)
Exchange Type The functionality of the exchange, for example, how messages are routed through it. Valid values are:
  • direct
  • fanout
  • topic
topic (not editable)
Reconnect Interval (Seconds)The amount of time, in seconds, between reconnection attempts. Default value is 5 seconds. This property is not displayed if AMQP - Lumada Default Setting is selected as the protocol.N/A
PersistentToggle to turn on persistent data.ON
Enable Store and ForwardWhen enabled, messages are written to disk when the connection is disrupted, and forwarded when the connection is restored.ON
Store and Forward Number of RetriesNumber of times to attempt forwarding.10
Store and Forward Notification IntervalInterval between forwarding attempts. Enter a number followed by ‘s’ for seconds, ‘h’ for hours, or ‘d’ for days.1s
Kafka protocol properties for Core
Property Description Lumada Default Setting
Hostname / IP Address The name of the host. IP-address
Port The port number to connect to the Kafka broker. 30092
Username The user name used for authentication with the Kafka broker. N/A
Password The password used for authentication with the Kafka broker. This property is not displayed if Kafka - Lumada Default Setting is selected as the protocol.N/A
Topic Category or feed name to which records are published. Kafka messages are organized in topics.N/A
Key (optional)Used by Kafka to partition the message.N/A
TimeoutDuration of the connection to Kafka.2s
Enable Store and ForwardWhen enabled, messages are written to disk when the connection is disrupted, and forwarded when the connection is restored.ON
Store and Forward Number of RetriesNumber of times to attempt forwarding.10
Store and Forward Notification IntervalInterval between forwarding attempts. Enter a number followed by ‘s’ for seconds, ‘h’ for hours, or ‘d’ for days.1s
MQTT protocol properties for gateway
PropertyDescriptionLumada Default Setting
TopicThe topic to publish or subscribe to. asset topic
Hostname / IP AddressThe host of the gateway device from which you want to collect data.gateway host
PortThe port for the gateway.gateway port
UsernameThe username required to log in to the gateway.N/A
PasswordThe password required to log in to the gateway.N/A
Quality of ServiceThe MQTT quality of service to use for message exchanges.

Options:

  • AtMostOnce (0)
  • AtLeastOnce (1)
  • ExactlyOnce (2)
ExactlyOnce (2)
QuiesceThe maximum number of seconds to wait for the gateway to pause before disconnecting.10 seconds
Dispatch methodMQTT or REST, for sending data from a gateway to IIoT Core. REST is selected automatically when the Data Size is set to large. The store and forward feature is supported by default for MQTT.MQTT

Data types

When creating a data route, these data types are available for your data profile.

Data typeFormat
HiotaThis data type uses the Common Data Format (CDF), a self-describing data format for storing scalar and multidimensional data in a platform- and discipline-independent way.
InfluxText-based format for writing points to a database.
JSONLanguage-independent, lightweight data interchange format that uses human-readable text to store and transmit data objects consisting of attribute-value pairs and array data types.
Parquet

Apache Parquet is an open source, columnar data file format that provides data compression, encoding schemes, and enhanced performance for moving data in bulk. In IIoT Core, it is used for submitting REST API calls for Machine learning service and digital twins.

NoteWhen using IIoT Core with Data Catalog, Parquet column names cannot contain special characters.
User DefinedAny supported MIME type.

Creating a data route

You can establish a data route from the data source to a data destination that connects through a supported protocol.

Create a data route using the IIoT Core Services UI

You can create a data route to a device by using the IIoT Core Services UI.

Before you begin

  • Create the asset for which you want to create a data route.
  • To receive data from a gateway device, add the gateway to the device inventory in the Devices window and provision it. If the gateway is in the Inventory state, you can add the gateway to the data route, but the gateway will not receive data until it is provisioned.
  • Verify the source and destination of the data route. For settings and default values, see Data sources.

Procedure

  1. In the IIoT Core Services UI, select the Data Route tab to open the Data Routes window.

  2. On the Data Routes window, click Create Data Route.

    The Create Data Route window is opened.
  3. In the Name and Asset section, enter a Name.

  4. Select an asset from the Asset list. The data that you collect is associated with the asset that you select.

  5. In the Device Type list, select from the following options:

    • Appliance (Core device).
    • Gateway
    • Device Group
  6. If you selected either the Gateway or Device Group device type, then select one of the following additional options:

    • For a Gateway device type, select a device from the Device list.
    • For a Device Group device type, select a group from the Device Group list.
  7. Enter a Trace ID. The default value for the Trace ID is the asset name.

  8. In the Data Profile section, select a Data Type.

    Consider the restrictions in the following table when selecting a data type:
    Data typeRestriction
    HIOTA

    Modbus only works with the HIOTA data type because it is built into the message format.

    REST ingress only works with the HIOTA or USER DEFINED data types because the endpoint either takes a HIOTA message as is or the data type is defined in the REST header.

    The HIOTA data type is not supported for small message optimization. (Batch selector in the Date Source section of the window.)

    INFLUX

    The INFLUX data type works with InfluxDB and MinIO databases and Kafka and AMQP protocols. The InfluxDB line protocol should not be written to CouchDB or Postgres.

    JSONN/A
    PARQUETParquet only works with REST as a data source and MinIO as a data destination.
    USER DEFINED

    The USER DEFINED data type cannot be saved to an InfluxDB data destination.

    REST ingress only works with the HIOTA or USER DEFINED data types because the endpoint either takes a HIOTA message as is or the data type is defined in the REST header.

    The USER DEFINED data type is not supported for small message optimization. (Batch selector in the Date Source section of the window.)

  9. Select one of the following Data Size options.

    • small (0B to 25KB)
    • medium (25KB to 1MB)
    • large (1MB to 25MB)
    CautionAfter the data size is submitted, the selection cannot be edited. When the Data Size is set too large, the dispatch method is automatically set to REST. The Data Size selection can affect the default values of the following data route configuration properties:
    • Quality of Service (MQTT and AMQP)
    • Timeout (InfluxDB)
    • Timeout (MinIO)
    • Idle Timeout (MinIO)
    • TLS Timeout (MinIO)
    • Expect Continue Timeout (MinIO)
    • Keep Alive (MinIO)
    • Max Idle Connections (MinIO)
    • Store and Forward Number of Retries
    • Store and Forward Notification Interval
  10. In the Data Source section, enter a Name for the data source.

  11. Select a protocol from the Protocol list.

    The following protocols are supported data sources for IIoT Core Services with the Appliance device type:
    ProtocolDescription
    AMQP

    Use the AMQP protocol as a data source for IIoT Core Services.

    AMQP supports batching of small messages.

    KafkaUse the Kafka protocol as a data source.
    RESTUse REST as a data source.
    The following protocols are supported data sources for IIoT Gateway with the Devices and Device groups device type:
    ProtocolDescription
    MQTT

    Use the MQTT protocol as a data source for the gateway.

    MQTT supports batching of small messages.

    ModbusUse the Modbus protocol as a data source for the gateway.

    For more information about data sources, see Data sources.

  12. In the Protocol field, in the Data Source section, if you select an option other than a default setting, complete the required fields to configure the protocol.

  13. Click More Options to update additional options.

    TipFor AMQP and MQTT, you can toggle the Batch selector to enable or disable small message optimization for the AMQP and MQTT protocols.
    • Off: Toggle off to disable batch processing for small messages. You can add additional data destinations.
    • On: Tpggle on to enable batch processing for small messages. You can set the Batch Count and Batch Timer fields by specifying the message size, how many messages should be batched together, and a timeout parameter. The ability to add additional data destinations is not available. The Batch Count needs to be equal to or smaller than QoS.
  14. In the Data Destinations section, enter a Name for the data destination.

  15. In the Destination Type list, select either protocol or database.

    NoteIf you enabled batch processing of small messages, only the InfluxDB database is supported as a destination.
  16. Select one of the following, additional options for the destination type that you selected:

    • For a protocol destination type, select a protocol from the Protocol list.
    • For a database device type, select a database from the Database Type list.
    See Data destinations for a description of values.
  17. In the Protocol field, in the Destination section, if you select an option other than a default setting, complete the required fields to configure the protocol.

  18. Click More Options to update additional options.

  19. If you specified Postgres as the database type, use one of the following options to update the Data Schema table:

    • Click Add, enter information for the Data Schema column, and then click Apply.
    • Click Import to import a database schema.
    NoteYou can also select the check box to Add hiota fields.
  20. Click More Options to update additional options.

  21. Click Add Another Data Destination to add another destination for the route.

    NoteYou cannot add additional data destinations if you enabled batch processing for an AMQP or a MQTT data source.
  22. Choose one of the following save options:

    • Click Save And Deploy to deploy the route.
    • Click Save to save the route as a draft.
    • Click Cancel to cancel creating the data route.

Results

After a data route is successfully deployed, the Status column on the Data Routes window is Deployed (Connected) for that data route.

Create a data route using REST as a data source

You can use REST to upload a binary object and send data to a preferred destination. You can send REST data to the following destinations:

  • AMQP
  • Kafka
  • MinIO
  • Postgres

Procedure

  1. Create a data route using REST as data source.

    For more information, see Create a data route using the IIoT Core Services UI.For the data route, you must know the following information when you send the binary file:
    • Trace ID
    • Content Type
    NoteREST only works with USER DEFINED or HIOTA data types.
  2. Send the binary file to https://<host>:30983/api/v1/ingress-rest.

Results

The file is routed to the destination selected when creating the data route.

View the details of the data route

You can view details for the data routes that are configured in the IIoT Core Services UI.

Procedure

  1. In the IIoT Core Services UI, click the Data Route tab .

    The Data Routes window opens.
  2. Navigate to the data route that you want to review.

  3. Click the actions menu icon GUID-860D1A4B-E960-45FE-81C1-4D6E31957DF8-low.png, and then select Details.

    A window opens with a title that matches the name of the data route.
  4. Click each of the following tabs on the window to view details about the data route:

    TabDescription
    DetailsThe route overview for the selected route for information about asset and device or device group, data profile, data source, and data destination.
    StatusThe status of the data source and data destinations of the selected route, such as Connected or Disconnected.
    AnalysisView and visualize data collected from data sources. For details on this tab, see Use data services.
    Store & ForwardView the number of records that are waiting to be forwarded. For details on this tab, see View store and forward data.
    NoteFor routes in draft state that are Incomplete or Ready to Deploy, only Details and Status tabs are displayed.
  5. (Optional): You can Edit, Stop, or Delete a data route.

    For instructions on editing, stopping, or deleting a data route, see:

Edit a data route

You can use IIoT Core Services to change the way a data route is configured.

Procedure

  1. In the IIoT Core Services UI, click the Data Route tab.

    The Data Routes window opens.
  2. Navigate to the data route that you want to edit.

    TipTo reduce the number data routes that appear in the table, you can click Filters, and then select options to filter the list.
  3. Click the actions menu icon GUID-860D1A4B-E960-45FE-81C1-4D6E31957DF8-low.png, and then click Edit.

  4. When you are done editing the data route, click one of the following options.

    • Save and Undeploy
    • Save and Deploy

Stop a data route

Stop a data route to temporarily halt the flow of data from a device to IIoT Core Services. You can stop a data route for maintenance or when the source device is not available. When a data route is stopped, data collection ceases, but data already collected is saved.

Procedure

  1. In the IIoT Core Services UI, click the Data Route tab to open the Data Routes window.

  2. Navigate to the route that you want to stop.

    TipTo reduce the number data routes that appear in the list, you can click Filters, and then select options to filter the list.
  3. Click the actions menu icon GUID-860D1A4B-E960-45FE-81C1-4D6E31957DF8-low.png, and then select Stop.

    When a data route is stopped, the status changes to Ready to Deploy.
  4. To re-deploy the data route, click Ready to Deploy.

Delete a data route

Delete a data route when you no longer collect data from an asset or when you want to modify a specified data size.

Before you begin

Make sure the data route either is not in use or the status is Deployed.

Procedure

  1. In the IIoT Core Services UI, click the Data Route tab to open the Data Routes window.

  2. Navigate to the data route that you want delete.

    TipTo reduce the number data routes that appear in the list, you can click Filters, and then select options to filter the list.
  3. Click the actions menu icon GUID-860D1A4B-E960-45FE-81C1-4D6E31957DF8-low.png, and then select Delete.

  4. In the Delete confirmation window, click Delete.

Results

The data route is deleted and removed from the Data Routes window.