Manage rules
Organizations need to understand the quality of their data to understand its fit for use. With Pentaho Data Catalog's rules framework, you can define, execute, and manage business rules.
Business rules creation on resource data
Define business rules to manage your data and ensure its quality. These rules allow you to set particular guidelines that you can set to measure the quality of your data by determining whether it meets specific standards and regulations. By defining these business rules, you can ensure that your organization's data is accurate, consistent, and reliable. You can define rules on the JDBC data source.
The user’s role determines the user's ability to create or execute rules.
- Data Quality Administrator: Lets a user define, review, and approve data quality rules and execute them.
- Data Quality Operator: Lets a user execute data quality rules and propose new rules but cannot approve new rules or edit existing ones.
- Data Quality Operator: Lets a user execute data quality rules and propose new rules but cannot approve new rules or edit existing ones.
Rule workflow
In Data Catalog, on the Business Rules page, you can create, update, edit, and delete rules.
Create a rule
Procedure
Click Management in the left navigation menu.
The Manage Your Environment page opens.Select Business Rules.
The Business Rules page opens.Click Add Business Rule.
In the Create Business Rule page, enter the following information.
Field Description Business Rule Name (Required) Enter the unique name of the rule that your users will recognize. Names must start with a letter, and contain only letters, digits, hyphens, or underscores. White spaces are supported, but trailing spaces are not allowed in names. Created by Select the username of the owner of the rule. The default value of this field is the logged-in user. Description Enter a description for this rule. For example, you may want to indicate the purpose of the rule to assist other users. Note Enter additional comments for the rule. For example, you may want to describe the workflow or use case of the rule. Rule Enabled By default, a new rule is enabled. Clear the check box to disable the rule. When a Rules Execution job is run, disabled rules are skipped and are not evaluated.
Rule Approved Select to approve the rule. This option is only available to users with the Data Quality Administrator role. Click Create Business Rule to save your rule.
In the confirmation window, click Configure to Configure a rule or click Close to configure it later.
Next steps
Configure a rule
Procedure
Locate the business rule you want to configure in the table of rules and select the View Details button (>) in its row.
If you have a large number of rules, select Show Filters to help you find the rule you want to edit.
The Business Rules page opens to the Details view for the selected rule.(Optional) Click Configuration tab and select the Business Rule Type.
The business rule type helps you differentiate between rules.(Optional) Select the Data Quality Dimension to differentiate between different quality rules that you create. Options are Accuracy, Uniqueness, Consistency, Timeliness, Conformity, Completeness, and Validity.
Set the schedule to run the business rule, select one of the following schedules:
- Daily
- Weekly
- Monthly
Set the rule scope and condition:
- Select resources on which you want the rule to be evaluated and applied. Select the target table or column.
- Define the rule's condition for evaluation using the SQL query. Write an SQL query to identify non-compliant data by returning rows that do not match compliant rows and the scope count.
In the following SQL example Customer table is selected to identify the customers with missing fax numbers.
SELECT count(*) total_count, count(c."Fax") scopeCount, SUM(CASE WHEN c."Fax" isnull THEN 1 ELSE 0 END) nonCompliant FROM chinook."Customer" c
total_count: The total number of rows in the Fax column in the Customer table.scopeCount: The number of customers with a fax number value in the Fax column.nonCompliant: The number of customers missing the value in the Fax column.In Configure Rule Actions, click Add Action, to add actions like Set Status, Apply Tags, and Webhook
Set the threshold at either a percentage or a specified row count and add the following action- Set Status: Set thresholds for PASS, WARNING, and FAIL status.
- Apply Tags: If the threshold is met the tags are applied to the resource.
- Webhook: Configure an external system capable of receiving HTTP requests. If the condition is met the extersyatem will receive a notification.
If the rule configuration values are entered correctly, click Save Changes.
If there is a problem while saving your rule, an error message appears indicating the problem. Fix the problem and save changes.To execute the rule, click Run Now at the top right of the page.
You can monitor the progress on the Workers page.
Results
Update a rule
Perform the following steps to edit a rule:
Procedure
Click Management in the left navigation menu.
The Manage Your Environment page opens.Select Business Rules.
The Business Rules page opens.Locate the business rule you want to configure in the rules table and select the View Details button (>) in its row.
If you have a large number of rules, select Show Filters to help you find the rule you want to edit.The Business Rule page opens to the Details view for the selected rule.Edit the fields as needed in the Details and Configuration views.
Click Save Changes.
The rule is saved with your changes. If there is a problem while creating your rule, an error notification will be displayed at the top of the page. Resolve the error and click Save Changes.
Delete a rule
Procedure
Click Management in the left navigation menu.
The Manage Your Environment page opens.Select Business Rules.
The Business Rules page opens.Click Management on the menu bar to open the Manage Your Environment page, and then select Business Rules.
The Business Rules page opens.Use the check box to select the rule you want to delete.
Click Remove.
In the confirmation message window, click Remove to delete the rule.
View rule execution information
You can access and view rule execution from the Business Rules page and the History page.
On the Business Rules page, you can find a list of all the rules that have been set up in the system. The list includes information about each rule, such as its name, description, execution status, and so on.
On the History page, you can view the information of the specific rule. This helps you to keep track of the status of the rule.
Procedure
Click Management in the left navigation menu.
The Manage Your Environment page opens.Select Business Rules.
The Business Rules page opens with a list of all the rules set up in the system.Locate the business rule to view details of a specific business rule and select the View Details button (>) in its row.
If you have a large number of rules, select Show Filters to help you find the rule you want to edit.The Business Rules page opens to the Details view for the selected rule.To view information on the selected rule, click the History tab, and then select the View Details button (>) in the row.
The View Details page opens.
Creating a rule group and assigning rules to a group
To organize the business rules, you can create a rule group and add your rules. This helps you keep your rules organized and easily accessible.
Procedure
Click Management in the left navigation menu.
The Manage Your Environment page opens.Select Rules Group.
The Business Rules Group page opens.Enter the Rule group name, Description, and Note.
Set the schedule to run all the business rules in a group, and select one of the following schedules: Daily, Weekly, and Monthly.
Click Assign Rules, select the rules you want to add to a group, and then click Assign.
Click Save Changes.