Skip to main content
Hitachi Vantara Lumada and Pentaho Documentation

Manage data routes

Control the flow of data from a device to a protocol or database.

You can create and manage a data route for a data source to one or more valid destinations. Destinations can be either protocols or databases.

A diagram of a data route, from source, to broker, to routing services, and then to a data destination either directly or through a broker and application.

Data route configuration values

The following tables describe the protocol and database options supported in IIoT Core Services.

Data sources

Gateways can receive data using the MQTT or Modbus protocol. IIoT Core can receive data through AMQP or REST.

AMQP protocol properties for Core
Property Description AMQP - Lumada Default Setting
Queue Name (Data Source only) The queue name to publish the message. Each asset must have a unique Queue Name.Specify the asset name appended with a serialized number and the word ‘queue.’
Hostname / IP Address The name of the host. IP-address
Port The port number to connect to the AMQP broker. 30671
Virtual HostThe virtual host path, if needed. Example: entering "mypath" becomes "<host>:<port>/mypath". This property is not displayed if AMQP - Lumada Default Setting is selected as the Protocol.N/A
Username The user name used for authentication with the AMQP broker. admin
Password The password used for authentication with the AMQP broker. This property is not displayed if AMQP - Lumada Default Setting is selected as the protocol.(Ask your administrator)
Binding Key The binding or routing key is a message attribute.

The data source Binding Key must match the Trace ID if the value for the Data Type parameter is application/json. If the Data Type value is application/x-hiota, the Trace ID must match the Trace ID in the Common Data Model. See Common Data Model.

Depending on the exchange type, the exchange might use this key to determine how to route messages to queues.

For Data Source, the value defaults to asset-name with the extension "route". For Data Destination, it defaults to asset-name-destination.

Quality of Service(Data Source only) The maximum amount of unacknowledged deliveries that are permitted on a channel. When the maximum number is reached, no more messages are delivered until at least one message is acknowledged. 10
Exchange Name Topic to publish or subscribe to depending on the direction of the route. hiota-exchange (not editable)
Exchange Type The functionality of the exchange, for example, how messages are routed through it. Valid values:
  • direct
  • fanout
  • topic
topic (not editable)
Reconnect Interval (Seconds)The amount of time, in seconds, between reconnection attempts. Default value is 5 seconds. This property is not displayed if AMQP - Lumada Default Setting is selected as the protocol.N/A
Allow Untrusted CA CertificatesSelf-signed certificates. Enable (default) to allow, disable to allow only trusted certificates. This property is not displayed if AMQP - Lumada Default Setting is selected as the protocol.N/A
Kafka protocol properties for Core
Property Description Kafka - Lumada Default Setting
Hostname / IP Address The name of the host. IP-address
Port The port number to connect to the Kafka broker. 30092
Username The user name used for authentication with the Kafka broker. N/A
Password The password used for authentication with the Kafka broker. This property is not displayed if Kafka - Lumada Default Setting is selected as the protocol.N/A
Topic Category or feed name to which records are published. Kafka messages are organized in topics.N/A
TimeoutDuration of the connection to Kafka.2s
MQTT protocol properties for gateway
PropertyDescriptionLumada Default Setting
TopicThe topic to publish or subscribe to. asset topic
Hostname / IP AddressThe host of the gateway device you want to collect data from.gateway host
PortThe port for the gateway.gateway port
UsernameThe user name required to log in to the gateway.N/A
PasswordThe password required to log in to the gateway.N/A
Quality of ServiceThe MQTT quality of service to use for message exchanges. Values are AtMostOnce (0), AtLeastOnce (1), or ExactlyOnce (2).ExactlyOnce (2)
QuiesceThe maximum number of seconds to wait for the gateway to pause before disconnecting.10 seconds
Dispatch methodMQTT or REST, for sending data from an IIoT Gateway to IIoT Core. REST is selected automatically when the Data Size is set to large. Store and forward is supported by default for MQTT.MQTT
Modbus protocol properties for gateway
Property Description Modbus - Lumada Default Setting
Hostname / IP Address The Modbus server host IP address. N/A
Host Port The server port number. The default port number is 502.N/A
Protocol The interface that Modbus supports.TCP
Operation Read operation. Read
Cardinality The start of the register or coil. N/A
Start Address The start address of the register or coil. N/A
Slave Address

Addresses of the devices supplying information.

Supports up to 247 secondary addresses.
N/A
Poll Interval Number of seconds between polls. 10 seconds
Data Descriptors The tags sent from Modbus, for example, Register [50] = Speed. N/A

Data destinations

The following tables describe the options for the database or protocol that you select as the Destination Type.

Couch database properties
Property Description Lumada Default Setting
Hostname / IP Address The name of the host on which the database resides. IP-address
Port The port number for the host. 30984, 30224 (Passport API)
Username The user name for the database connection. admin
Password The password for the user name. This property is not displayed if COUCH - Lumada Default Setting is selected. (Ask your administrator)
Database NameThe asset selected during route creation. Each asset must have a unique queue name.asset name with the extension "db"
Allow Untrusted CA CertificatesSelf-signed certificates. Select to allow, deselect to allow only trusted certificates. This property is not displayed if COUCH - Lumada Default Setting is selected.N/A
Save IT Data (Optional) This causes fields to be saved in the IT Data section of the CDM. Disabled
Enable Store and ForwardWhen enabled, messages are written to disk when the connection is disrupted, and forwarded when the connection is restored.Yes
Store and Forward Number of RetriesNumber of times to attempt forwarding.10
Store and Forward Notification IntervalInterval between forwarding attempts. Enter a number followed by ‘s’ for seconds, ‘h’ for hours, or ‘d’ for days. 1s
Influx database properties
Property Description Lumada Default Setting
Hostname / IP Address The name of the host on which the database resides. IP-address
Port The port number for the host. 30086, 30223 (Passport API)
OrganizationA way for Influx to organize data.N/A
TokenSecurity token needed to access the Influx database.N/A
BucketName of the Influx database section used.<asset-name>_db_<number>
Retention PolicyThe length of time to retain messages. Enter a number followed by ‘s’ for seconds, ‘h’ for hours, or ‘d’ for days. No data is purged if the value is left blank.N/A
Timeout The number of seconds requests wait for the client to complete a database transaction. 1s
PrecisionHow frequently the Influx database saves the data.nanoseconds
Save IT Data (Optional) This causes fields in the IT Data section of the CDM to be saved. Disabled
Allow Untrusted CA CertificatesSelf-signed certificates. Select to allow, deselect to allow only trusted certificates. This property is not displayed if the INFLUX - Lumada Default Setting is selected.Enabled
Enable Store and ForwardWhen enabled, messages are written to disk when the connection is disrupted, and forwarded when the connection is restored.Enabled
Store and Forward number of retriesNumber of times to attempt forwarding.10
Store and Forward Notification IntervalInterval between forwarding attempts. Enter a number followed by ‘s’ for seconds, ‘h’ for hours, or ‘d’ for days. 1s
MinIO database properties
Property Description Lumada Default Setting
Hostname / IP Address The name of the host on which the database resides. IP-address
Port The port number for the host. 31000
Access Key The MinIO access key. Automatically generated or defined during installation.
Secret Key The MinIO secret key. This property is not displayed if MINIO - Lumada Default Setting is selected. (Ask your administrator)
Bucket The asset selected during route creation. This property can only contain lowercase alphanumeric characters or dashes and must start with a letter.asset name
Timeout The timeout duration in seconds when trying to connect to the database.10
Idle Timeout The timeout duration for requests, in seconds.10
TLS Timeout The timeout duration in seconds for TLS handshakes. 10
Expect Continue Timeout The expect continue timeout, in seconds. 10
Keep Alive The keep alive time for requests, in seconds. 10
Max Idle Connections The maximum number of idle connections that can occur. 10
Allow Untrusted CA CertificatesSelf-signed certificates. Enable (default) to allow, disable to allow only trusted certificates. This property is not displayed if MINIO - Lumada Default Setting is selected.N/A
Compression Database compression. Enabled
Enable Store and ForwardWhen enabled, messages are written to disk when the connection is disrupted, and forwarded when the connection is restored.Yes
Store and Forward Number of RetriesNumber of times to attempt forwarding.10
Store and Forward Notification IntervalInterval between forwarding attempts. Enter a number followed by 's' for seconds, 'h' for hours, or 'd' for days.1s
Postgres database properties
Property Description
ColumnName of the column. Each column has a unique name.
TypeData types for the column:
  • smallint
  • int
  • bigint
  • bool
  • varchar
  • real
  • float
  • timestamptz
  • bytea
  • text
  • json
  • uuid
SizeMaximum data size for the column. Only applies for varchar and binary.
ConstraintsThe following constraints are supported:
  • primary Key
  • not null
  • unique
  • default (n)
Hostname / IP AddressThe host IP address where Postgres resides.
PortPort number of the host.
UsernameUser name to log in to the host.
PasswordPassword to log in to the host.
DatabaseDatabase name.
TableTable name in the database to store the data.
Store and Forward number of retriesNumber of times to attempt forwarding.
Store and Forward notification intervalInterval between forwarding attempts.
AMQP protocol properties for Core
Property Description Lumada Default Setting
Hostname / IP AddressThe name of the host. IP-address
Port The port number to connect to the AMQP broker. 30671
Virtual Host

The virtual host path, if needed.

Example: Entering "mypath" generates "<host>:<port>/mypath".

This property is not displayed if AMQP - Lumada Default Setting is selected as the Protocol.

N/A
Username The user name used for authentication with the AMQP broker. admin
Password The password used for authentication with the AMQP broker. This property is not displayed if AMQP - Lumada Default Setting is selected as the protocol.(Ask your administrator)
Binding Key The binding or routing key is a message attribute.

The data source Binding Key must match the Trace ID if the value for the Data Type parameter is application/json. If the Data Type value is application/x-hiota, the Trace ID must match the Trace ID in the Common Data Model. See Common Data Model.

Depending on the exchange type, the exchange might use this key to determine how to route messages to queues.

For Data Source, the value defaults to asset name with the extension "route". For Data Destination, it defaults to asset name destination.

Exchange Name Topic to publish or subscribe to depending on the direction of the route. hiota-exchange (not editable)
Exchange Type The functionality of the exchange, for example, how messages are routed through it. Valid values are:
  • direct
  • fanout
  • topic
topic (not editable)
Reconnect Interval (Seconds)The amount of time, in seconds, between reconnection attempts. Default value is 5 seconds. This property is not displayed if AMQP - Lumada Default Setting is selected as the protocol.N/A
PersistentToggle to turn on persistent data.ON
Enable Store and ForwardWhen enabled, messages are written to disk when the connection is disrupted, and forwarded when the connection is restored.ON
Store and Forward Number of RetriesNumber of times to attempt forwarding.10
Store and Forward Notification IntervalInterval between forwarding attempts. Enter a number followed by ‘s’ for seconds, ‘h’ for hours, or ‘d’ for days.1s
Kafka protocol properties for Core
Property Description Lumada Default Setting
Hostname / IP Address The name of the host. IP-address
Port The port number to connect to the Kafka broker. 30092
Username The user name used for authentication with the Kafka broker. N/A
Password The password used for authentication with the Kafka broker. This property is not displayed if Kafka - Lumada Default Setting is selected as the protocol.N/A
Topic Category or feed name to which records are published. Kafka messages are organized in topics.N/A
Key (optional)Used by Kafka to partition the message.N/A
TimeoutDuration of the connection to Kafka.2s
Enable Store and ForwardWhen enabled, messages are written to disk when the connection is disrupted, and forwarded when the connection is restored.ON
Store and Forward Number of RetriesNumber of times to attempt forwarding.10
Store and Forward Notification IntervalInterval between forwarding attempts. Enter a number followed by ‘s’ for seconds, ‘h’ for hours, or ‘d’ for days.1s
MQTT protocol properties for gateway
PropertyDescriptionLumada Default Setting
TopicThe topic to publish or subscribe to. asset topic
Hostname / IP AddressThe host of the gateway device from which you want to collect data.gateway host
PortThe port for the gateway.gateway port
UsernameThe username required to log in to the gateway.N/A
PasswordThe password required to log in to the gateway.N/A
Quality of ServiceThe MQTT quality of service to use for message exchanges.

Options:

  • AtMostOnce (0)
  • AtLeastOnce (1)
  • ExactlyOnce (2)
ExactlyOnce (2)
QuiesceThe maximum number of seconds to wait for the gateway to pause before disconnecting.10 seconds
Dispatch methodMQTT or REST, for sending data from a gateway to IIoT Core. REST is selected automatically when the Data Size is set to large. The store and forward feature is supported by default for MQTT.MQTT

Data types

When creating a data route, these data types are available for your data profile.

Data typeFormat
HiotaThis data type uses the Common Data Format (CDF), a self-describing data format for storing scalar and multidimensional data in a platform- and discipline-independent way.
InfluxText-based format for writing points to a database.
JSONLanguage-independent, lightweight data interchange format that uses human-readable text to store and transmit data objects consisting of attribute-value pairs and array data types.
ParquetApache Parquet is an open source, columnar data file format that provides data compression, encoding schemes, and enhanced performance for moving data in bulk. In IIoT Core, it is used for submitting REST API calls for Machine learning service and digital twins.
User DefinedAny supported MIME type.

Creating a data route

You can establish a data route from the data source to a data destination that connects through a supported protocol.

Create a data route using the IIoT Core Services UI

You can create a data route to a device in the IIoT Core Services UI.

Before you begin

  • Verify that the asset that you want to create a data route for is created.
  • To receive data from a gateway device, verify that the gateway is added to the device inventory and is provisioned. If the gateway is in Inventory state, you can still add it to the data route, but it will not receive data until it is provisioned.
  • You have identified the source and destination of the data route. For settings and default values, see Data sources.

Procedure

  1. In the IIoT Core Services UI, select the Data Route tab to enter the data route inventory.

  2. Click the Create Data Route button.

    The Create Data Route page opens.
  3. In the Name field, enter a name for the data route.

  4. In the Asset field, select an asset. The data you collect is associated with this asset.

  5. Under Device Type, select from the following:

    • Appliance (core).
    • Gateway: If the device type is a gateway, use the Device list to select a device in the inventory.
    • Device Group: If the device type is a device group, use the Device Group list to select a device group in the inventory.
  6. If you selected Gateway or Device Group in the Device Type field, you will be asked to select the device or device group in the corresponding Device or Device Group field.

  7. Enter a Trace ID. The Trace ID uses the asset name as default.

  8. Choose a data profile.

    1. Select the Data Type, taking the following restrictions into consideration.
      Data typeRestriction
      HIOTA

      Modbus only works with the HIOTA data type because it is built into the message format.

      INFLUX

      This works with Influx, Kafka, AMQP, and MinIO. The Influx line protocol should not be written to Couch or Postgres.

      JSONN/A
      PARQUETParquet only works with REST as a data source and MinIO as a data destination.
      USER DEFINED

      The USER DEFINED data type cannot be saved to an Influx data destination.

      Not supported for small message optimization. (Batch selector in the data source step)

      HIOTA or USER DEFINED

      REST ingress only works with these data types. This is because the endpoint either takes a HIOTA message as is or the data type is defined in the REST header.

      Not supported for small message optimization. (Batch selector in the data source step)

    2. Select the Data Size.
      • small (0B to 25KB)
      • medium (25KB to 1MB)
      • large (1MB to 25MB)
    When the Data Size is set to large, the dispatch method is automatically set to REST. Your Data Size selection can affect the default values of the following data route configuration properties:
    • Quality of Service (MQTT and AMQP)
    • Timeout (Influx)
    • Timeout (Minio)
    • Idle Timeout (Minio)
    • TLS Timeout (Minio)
    • Expect Continue Timeout (Minio)
    • Keep Alive (Minio)
    • Max Idle Connections (Minio)
    • Store and Forward Number of Retries
    • Store and Forward Notification Interval

    Once submitted, the data size cannot be edited.

  9. Choose a Data Source.

    1. Specify a name for the data source.

    2. Select a protocol, either using default settings or a custom setting.

      Supported data sources for IIoT Core Services:
      ProtocolDescription
      AMQP

      Use the AMQP protocol as a data source for IIoT Core Services.

      NoteAMQP supports batching of small messages.
      RESTUse REST as a data source.
      KafkaUse the Kafka protocol as a data source.
      Supported data sources for IIoT Gateway:
      ProtocolDescription
      MQTT

      Use the MQTT protocol as a data source for the gateway.

      NoteMQTT supports batching of small messages.
      ModbusUse the Modbus protocol as a data source for the gateway.
      For more information about data sources, see Data sources.
    (Optional) For AMQP and MQTT, you can enable small message optimization by turning on the Batch selector:
    • Off: No batch processing for small messages. You can add additional data destinations.
    • On: You can now set the Batch Count and Batch Timer fields by specifying the message size, how many messages should be batched together, and a timeout parameter. The ability to add additional data destinations is not available. The Batch Count needs to be equal to or smaller than QoS.
  10. Choose Data Destinations.

    1. Specify a name for the data destination.

    2. Select a destination type of either protocol or database.

    3. Enter values for the protocol or database type. See Data destinations for a description of values.

    4. When Postgres is the data destination, add and describe each column, or you can import a database schema.

    NoteIf you enabled batch processing of small messages, only the Influx database is supported as a destination.
  11. (Optional) Click Add Another Data Destination to add another destination for the route.

    NoteYou cannot add additional data destinations if you enabled batch processing for an AMQP or a MQTT data source.
  12. Click Save to save as a draft or Save And Deploy to deploy the route.

Results

When the data route has been successfully deployed, the Status column in the data route inventory shows Deployed (Connected).

Create a data route using REST as a data source

You can use REST to upload a binary object and send data to your preferred destination. You can send REST data to the following destinations:

  • AMQP
  • Kafka
  • MinIO
  • Postgres

Procedure

  1. Create a data route using REST as data source.

    For more information, see Create a data route using the IIoT Core Services UI.For the data route, you need to know the following information when you send the binary file:
    • Trace ID
    • Content Type
    NoteREST only works with USER DEFINED or HIOTA data types.
  2. Send your binary file to https://<host>:30983/api/v1/ingress-rest.

Results

The file is routed to the destination selected when creating the data route.

View the details of the data route

You can see and manage the data routes that are configured in the IIoT Core Services UI.

In the IIoT Core Services UI, select the Data Route tab to see the data route inventory. For each data route, you can perform the following actions:

  • Select a data route to open a tabbed view for that route:
    • Details: The route overview for the selected route for information about asset and device or device group, data profile, data source, and data destination.
    • Status: The status of the data source and data destination(s) of the selected route, such as Connected or Disconnected.
    • Analysis: View and visualize data collected from data sources. For details on this tab, see Use data services.
    • Store & Forward: View the number of records that are waiting to be forwarded. For details on this tab, see View store and forward data.
    NoteFor routes in draft state that are Incomplete or Ready to Deploy, only Details and Status tabs are displayed.

    You can also edit, stop, or delete the selected route using the Edit, Stop, or Delete buttons above the tabs.

  • Select the Actions menu ( GUID-860D1A4B-E960-45FE-81C1-4D6E31957DF8-low.png ) for a particular data route to edit, delete, or stop the route, or to view route details.

The following table describes the possible data route statuses and actions you can take for each.

StatusDescriptionAction
Ready to deployThe data route is complete, with all required information. It is saved as a draft and is not active.Click Ready to Deploy to deploy the data route.
Deployed (Connected)The data route is deployed successfully, and IIoT Core Services is receiving and sending data.N/A
Deployed (Warning)The data route is deployed but is not receiving data.N/A
Deployed (Error) The data route is deployed but has encountered an error.

Select the route and go to the Status tab for more information about the error.

For information about the root cause of errors, view the log in the Administration Deployment tab.

Incomplete Required values are missing from the data route. It is saved as a draft and remains inactive.Click Edit from the Actions menu to complete the data route.

Edit a data route

You can use IIoT Core Services to change the way a data route is configured.

Procedure

  1. Perform the following steps to edit a data route:
  2. In the IIoT Core Services UI, select the Data Route tab to enter the data route inventory.

  3. To edit a data route, select Edit from the Actions menu ( GUID-860D1A4B-E960-45FE-81C1-4D6E31957DF8-low.png ) next to the route name in the data route inventory.

    Alternatively, to locate a particular data route, use one of the filters above the data route inventory table.
  4. When you are done editing the data route, click Save and Undeploy or Save and Deploy to update the data route properties.

Stop a data route

You can use the IIoT Core Services UI to temporarily halt the flow of data from a device to IIoT Core Services. You can stop a data route for maintenance or when the source device is not available. When a data route is stopped, data collection ceases, but data already collected is saved.

Procedure

  1. Perform the following steps to stop a data route:
  2. In the IIoT Core Services UI, select the Data Route tab to enter the data route inventory.

  3. To stop a data route, click Stop from the Actions menu ( GUID-860D1A4B-E960-45FE-81C1-4D6E31957DF8-low.png ) next to the route name in the data routes inventory.

  4. When a data route is stopped, the status changes to Ready to Deploy.

    To re-deploy the data route, click Ready to Deploy.

Delete a data route

You can use IIoT Core Services to delete a data route to no longer collect data from an asset or when you want to modify a specified data size.

Before you begin

Make sure the data route is not in use or has a Deployed status.

Procedure

  1. Perform the following steps to delete a data route:
  2. In the IIoT Core Services UI, select the Data Route tab to enter the data route inventory.

  3. To delete a data route, select Delete from the Actions menu ( GUID-860D1A4B-E960-45FE-81C1-4D6E31957DF8-low.png ) next to the route name in data routes inventory.

  4. In the Delete confirmation window, click Delete to confirm or click Cancel to keep the data route.

Results

The data route is removed from the data route inventory.