Manage alerts

Alerts Overview

Alerts are displayed on the Alerts tab.

The Alerts tab shows alerts that have been triggered by conditions that are built in to IIoT Core Services (out-of-box alerts). Alerts can also be triggered by custom applications that you build and deploy in IIoT Core Services.

To check for new alerts, refresh the Alerts page.

NoteThe time stamp displayed with an alert shows the time that processing of the alert is completed, not the time that the alert event is triggered.

Out-of-box alerts

IIoT Core Services has built-in (out-of-box) alerts to help monitor your system.

It generates an alert when certain conditions are met. The following table shows conditions that generate out-of-box critical alerts. They apply to both core and gateway devices.

Alert item	Alert name	Critical alert trigger	Description
CPU usage	HighCPU	>75%	More than 75% of the CPU capacity of a node is being utilized. The node is identified in the alert message.
Disk space	LowDiskSpace	>75%	More than 75% of disk space of a node is full. The node is identified in the alert message.
Memory usage	HighMemory	>75%	More than 75% of the RAM of a node is in use. The node is identified in the alert message.
Service restart	ServiceCrashLooping	>10 restarts within 15 minutes	One of the IIoT Core Services services has restarted 10 times within a span of 15 minutes. The service is indicated in the alert message.
Service down	ServiceNotReady	>15 minutes of downtime	One of the IIoT Core Services services has not been running for a span of at least 15 minutes. The service is indicated in the alert message.

Checking for new alerts

To check for new alerts while you are in the Alerts tab, refresh the page.

Obtaining AssetID for sending alerts

To send an alert message from your application, you need to set the AssetID for your application. You can retrieve the AssetID from the Core server, but if the gateway connectivity is intermittent, you may encounter difficulties.

On the Core server, Hiota-asset provides the AssetID.

On the gateway, the AssetID is set to the environment variable HIOTA_ASSET_ID. The AssetID does not change once it is provisioned. Because all the messages derive from the same gateway, there is only this one AssetID for all alerts.

Managing resource limits for container CPU and memory

The CPU and memory usage of a container is based on usage measured by the cadvisor and on the resource limits specified in a pod. Resource limits can be configured in a Kubernetes YAML file the same way as it is done for the Core server. See Deploy software configurations or secrets on the Core device.

If resource limits are not specified by a user, these default values apply:

CPU: 1
Memory: 512MB

If a container attempts to use more resources than is allocated, then the container is killed and restarted to prevent a processing error in the host system.

To change this behavior, use the following setting:

device.oom-kill-disable: true

This corresponds to the Docker runtime option --oom-kill-disable.

Setting resource request

If requests (a reserved resource) are omitted, Kubernetes automatically sets the CPU and memory usage in requests to the limit values. If the node doesn't have enough resources, the pod goes into pending mode so as not to break the system.

Create custom alerts

You can create custom alerts in your applications for any node, asset, or other aspect of your IIoT Core Services deployment.

Alerts triggered by conditions that you set appear in the Alerts tab.

The following process is a high-level framework for creating custom alerts:

Procedure

Write an application to send the alert, using the Common Data Model (CDM).
See related information in these sections:
- Common Data Model
- The Binding Key section of Data route configuration values
Deploy the app.
To deploy to IIoT Core Services, see Deploy container software in IIoT Core Services. To deploy to an IIoT Gateway, see Deploying container software on a gateway.
Publish the alert, in the CDM, to the hiota-alerts binding key.

Results

Alerts are displayed in the Alerts tab. Select the Alerts tab to see all alerts that have been triggered, including built-in (out-of-box) alerts.

Using the notification service

You can configure email notifications to issue alerts.

Set notification preferences

You can use notification preferences for devices to set how notifications are issued. You can choose to be notified for all devices or only be notified for those that are bookmarked.

Before you begin

Set up email notifications for alerts. See Set up email notifications.

Procedure

In the IIoT Core Services UI, select the Device tab.
Click the Notification Settings button to open the Notification Settings window.
Under Allow Notifications, select to receive notifications for All devices, Bookmarked devices only, or Disable .
To bookmark a device, see Bookmark a device.
In the Device Notifications section, set Device Email Notifications by using the selector.

NotePush notification are not supported in this release.
Click Apply.

Results

Notification preferences have now been set.

Set up email notifications

Before you begin

You must have a Kubernetes dashboard installed to configure this notification option.

Use this service to set up email notifications for alerts. You can develop custom email notifications by creating an SMTP plugin that takes alert information received through AMQP and converts it for SMTP.

Procedure

Use the SMTP plugin for configuring notifications provided by IIoT Core Services.
Go to your Kubernetes dashboard.
Select hiota in the Namespace list in the left-hand navigation.
In the Config and Storage section, select the Config Maps option.
For the config map hiota-notification-plugin-config, select the Actions menu ( ) and then View/edit YAML.
In the data section of the config map, add or overwrite the config.yaml with the following values:
```
config.yaml:
  - type: email
    enabled: true
    profile: "smtp"
```

Locate the config map hiota-notification-plugin-profile, select the Actions menu ( ) and then View/edit YAML.

In the data section of the config map, add or overwrite the profile section with the following values:

- profileName: 'smtp'
  properties:
    hostname: '<SMTP server>'
    port: 587
    auth: 'PlainAuth'
    identity: '' # Optional
    sender: <sample>@<domain>.com # This must be an email address. If this is not set, the value is SMTP_USER.

Back in the Config and Storage section in the navigation pane, select Secrets to set the user name and password.
Locate the secret hiota-notification-smtp-secrets, select the Actions menu ( ) and then View/edit YAML.
1. Scroll down to the data setting for user name and password information.
```
  "data": {
    "SMTP_PASSWORD": "",
    "SMTP_USER": ""
  },
```
2. Change data to stringData.
3. Enter values for SMTP_USER and SMTP_PASSWORD.
4. Click UPDATE.
  Restart the hiota-notification pod by deleting the pod and letting Kubernetes reschedule a new one using the following command:
```
kubectl delete pod hiota-notification-<uuid> -n hiota
```
In the CDM, publish the alert to the hiota-alerts or hiota-notification binding key.

Results

The alert is sent to the designated email address.

See the following JSON payload example.

For additional information, see:

Common Data Model
The Binding Key property in the AMQP sections of Data route configuration values

JSON payload example:

{
  "traceId": ["hiota-alerts"],
  "assetInfo": {
    "urn": "d8d38975-ffe2-4b8e-9a46-b53be80bd633",
    "name": "device"
  },
  "alertData": {
    "name": "Critical Alert",
    "description": "device has exceeded CPU threshold",
    "severity": "ALERT_SEVERITY_CRITICAL",
    "type": "ALERT_TYPE_HIGH_CPU",
    "notifications": [{
      "notificationType": {
        "emailNotify": {
          "subject": "Notification from NiFi",
          "bodyContentType": "text/plain",
          "body": "This is a notification test message sent from NiFi", "recipients": {
            "to": ["lumada-edge@outlook.com"],
            "cc": ["other-recipient@domain.com"]
          }
        }
      }
    }]
  }
}

Archive alerts

You can clean up the alerts repository in IIoT Core Services by archiving select alerts.

Procedure

In the IIoT Core Services UI, navigate to the Alert tab.
Select one or more alerts to archive.
- Individual alert: Click the Archive button for the corresponding row.
  NoteThe alert is immediately archived and disappears from the list.
- Multiple alerts: Use the check boxes to select any number of alerts and click the Archive button above the list. In the confirmation window, click the Archive button to confirm.

Results

The selected alerts are archived and removed from view.