Troubleshooting
This section explains how to resolve some typical issues that you might encounter.
This list provides common issues. See the IIoT Core Release Notes for known issues and workarounds.
Subscribe to errors
Subscribe to errors to preemptively catch and handle any errors as they occur.
Errors from the message broker are sent to the error routing (binding) key hiota-errors.
Use the RabbitMQ user interface for error subscription. Include the following information to subscribe to hiota-errors:
- URL: https://<cluster_fqdn>:31671
- Login: admin
- Password: (Ask your administrator)
- Host: <IP-address>
- Port: 30671
- Error Routing (Binding) Key: hiota-errors
- Exchange Name: hiota-exchange (default)
Troubleshoot applications
You can use logging and monitoring to troubleshoot your IIoT Core Services applications.
The centralized logging capability gives insight into the state of the IIoT Core Services components.
The current version of IIoT Core Services supports CLI. You can also use a text editor, or GUI tools such as Kibana (when Elasticsearch is installed) for logging. Deploy GUI tools as custom applications in the tab of the IIoT Core Services UI.
Verify that the IIoT Core Services are working by going to the Device tab in the IIoT Core Services UI. Select the IIoT Core Services device and open the Device Details page.
All the listed services should be running.
When you have Elasticsearch installed, you can use the curl command to query the service by the port. The unified log (UL) is exposed via port 30920 on the cluster. It is protected by the user name and the password that is set during installation. For this example, the user name is ‘admin’ and the password is ‘Admin123’:
# curl -k -u admin:Admin123 'https://<IP address of host>:30920/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open logstash-2018.10.26 CmcAfaevTk2Hv2L_VGa5sA 5 1 49004 0 46.1mb 20.9mb
You can further query the logs for errors or other information. Basic queries can be done using query string parameters in the URL. For example, the following searches for the text error in any field in any document and returns at most 5 results. Basic queries use the q query string parameter that filters on specific fields, (for example, fieldname:value), wildcards (abc*), and others. Other options such as size, from, and so on, can further customize the query and its results. For more information, see the Elasticsearch URI request documentation: http://www.elasticsearch.org/guide/reference/api/search/uri-request.html
# curl -k -u admin:Admin123 'https://<IP address of host>:30920/_search?q=error&size=5&pretty'
...
{
"_index" : "logstash-2018.10.26",
"_type" : "flb_type",
"_id" : "Qd3CsWYB7jqeXHKt-Zm9",
"_score" : 5.9804873,
"_source" : {
"es_time" : "2018-10-26T19:03:33.102Z",
"log" : "2018-10-26 19:03:33.102 [INFO][93] route_table.go 403:
Failed to get interface; it's down/gone. error=Link not found ifaceName=
\"cali329675177da\" ipVersion=0x4\n",
"stream" : "stdout",
"time" : "2018-10-26T19:03:33.102633914Z"
}
...
Access gateway logs
You can obtain a log of the services running on a gateway.
You can use the gateway log to troubleshoot problems on the gateway.
Before you begin
) menu is not available. Logs can only be collected when the gateway is online.Procedure
In the IIoT Core Services UI, select the Device tab.
Select Logs from the Actions menu (
) next to the device description.Select Request New Log.
Establish a time frame by selecting a time or date number, and then select the up or down arrow to adjust the number.
Select Request.
When the status shows Success, select Download Log.
Change log rotation in IIoT Gateway
To ensure that the IIoT Gateway software is able to run in a healthy and continuous way, the gateway log rotation is turned on by default.
The log rotation is configured as follows:
| Property | Value |
| enableDockerLogRotation | true |
| dockerLogMaxFiles | 7 |
| dockerLogMaxSize | 100 |
| log level | Info |
Because the default maximum number of Docker log files is 7 and the maximum size of each log file is 100 MB, by default at most 700 MB disk space can be consumed by Docker logs.
You can change the default values for log rotation on the IIoT Gateway. Use SSH to connect to the gateway and edit the following file: /etc/hiota/agent/configs/configuration.yaml.