Managing rules

Parent Topic

• Rules creation

Parent Topic

• Rules creation

Child Topics

• Rule criteria on resource data
• Rule criteria on resource metadata
• Rule criteria requirements and syntax

A rule action can perform an array of tasks. Actions can be one of the following:

• Set Properties
• Reset Properties
• Add Business Terms
• Remove Business Terms
• Set Term Assignment State
• Set Sensitivity
• Set Quality Dimension
• Compute Data Quality Score

You can define the rule action by specifying action types in the Rule Actions window. The supported rule action types are described below.

• Add Business Terms

  When actionType is set to AddBusinessTerm, the ruleAction makes term associations based on rule evaluation. You can apply a term suggestion on a specific field or on a qualifying resource. When applying a term suggestion of a field, the field is identified with a full field name. You can add multiple terms in AddBusinessTerms rules.

  NoteThe Data Catalog rule framework does not create new terms. Any term suggestions to be applied as part of a rule action must be for existing terms. If an associated term does not exist, Data Catalog displays an error message.

• Remove Business Terms

  When actionType is set to RemoveBusinessTerm, the ruleAction removes the term associations based on the rule evaluation.

• Set Properties and Reset Properties

  When actionType is set to SetProperties or ResetProperties, the ruleAction sets or resets custom property values.

  Property values are strings. If you specify property names with @@, then the value of its string is substituted for the property name. You can specify multiple property values in the Add Property Value section. Each property value should be comma (,) separated and without any trailing space.

  You can use property actions to set and reset property values. To reset a property value, use ResetProperties.

• Set Quality Dimension

  When actionType is set to SetQualityDimension, the ruleAction defines the quality dimension. You can specify the following quality dimensions with a threshold value range (Low Threshold and High Threshold).

  The supported quality dimensions are:

  • Accuracy
  • Completeness
  • Consistency
  • Timeliness
  • Uniqueness
  • Validity

  Specify the field name if the associated action is to be performed at the field level. If you do not specify the field name, then the action is performed at the resource level.

  NoteThis action requires non-metadata criteria to execute because it evaluates the actual data of the column.

• Set Term Assignment State

  When actionType is set to SetTermAssignmentState, the ruleAction updates the business term assignment state. You can specify one or multiple terms whose assignment state is to be changed. If you do not select a term, then the selected assignment state is applied to all the terms associated with the resource.

  The supported business term assignment states are:

  • Accept & Use In Learning
  • Accept & Don’t Use In Learning
  • Reject & Use In Learning
  • Reject & Don’t Use In Learning

  Use In Learning is only applicable at the column level. At the resource level, it can only be Accept or Reject.

  NoteThis action requires metadata criteria in the rule to execute.

• Compute Data Quality Score

  When actionType is set to ComputeDataQualityScore, the ruleAction computes the data quality score based on the provided data quality dimension values.

  When you select the Compute Data Quality Score as the rule action type, the template for the score formula is provided in the Score Formula field. You can customize the template formula by changing the dimension weightage values.

  Score formula template:

  

  Data quality score formula conditions:

  • The column-level entity should have at least one calculated dimension for the action to execute.
  • If the rule provides weightages to multiple dimensions, and the actual entity has only one dimension, then 0 is considered the value for the other dimensions.

  • The formula should have at least one dimension and all the weights provided should total 1.0.

    The following example shows a valid formula with weights specified for Accuracy, Completeness and Uniqueness quality dimensions:

    

  Data quality score calculation process:

  Data quality score is calculated based on the Quality Dimension Weightage specified in the formula and Calculated Value Set defined for the quality dimension.

  Data quality score calculation process is stated below with an example:

  Data quality score formula:

  • Accuracy*0.5 + Completeness*0.20 + Uniqueness*0.30

  Calculated value set:

  • Accuracy: 60
  • Completeness: 80
  • Uniqueness: No calculated value is set for Uniqueness. The default value for Uniqueness will be 0.

  Data quality score calculation:

  • [60*0.5 + 80*0.2 + 0*0.3] = 46
  NoteThis action requires a metadata criteria in the rule to execute.

• SetSensitivity

  When actionType is set to SetSensitivity, the ruleAction updates the resource sensitivity level. This is applicable only at the resource level. If you do not provide the resource name, then the action changes the sensitivity level of all the resources defined in the rule scope and criteria.

  If multiple sensitivity rules are executed for a particular resource, then the sensitivity level defined in the latest rule is considered, overriding all the previous rules.

  The supported business term assignment states are:

  • Low
  • Medium
  • High
  • Non Sensitive
  • Unknown
  NoteThis action requires metadata criteria in the rule to execute.

Parent Topic

• Rules creation

Procedure

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.

2. Select Business Rules.

   The Business Rules page opens.

3. Click Add Business Rule.

   The Create Business Rule page opens.

4. Enter the rule name and description.

5. Set the Rule Scope. The rule scope includes the following parameters:

   You can create your own scope and save it for future use by yourself or another user or select one of the existing ones using the Load Scope option.

   You can use the Rule Scope JSON box to view the rule scope as you create it or the rule scope that you have selected using the Load Scope option. This helps you visualize the defined rule scope parameters and make the necessary changes. You can also copy the rule scope in the JSON format and use it for REST API requests.

   • Virtual Folders

     Select virtual folders on which to execute the rule. You must include at least one virtual folder for the rule to compile. When a single virtual folder is listed, the rule executes against all the resources in the listed virtual folder. You can select additional virtual folders. If a folder no longer exists when the rule is executed, it is ignored.

   • Resource Terms

     Select business terms to filter the virtual folder resources.

   • Field Terms

     Select the business terms associated with the fields or columns to filter the virtual folder resources.

   • Custom Properties

     Select the custom properties to filter the virtual folder resources.

   • Term States

     You can further filter the virtual folder resources based on the term association state. The possible values are Accepted, Rejected, and Suggested. If you do not specify a term association state, all states are included.

6. Set the Rule Criteria.

   You can create your rule criteria and save it for future use by yourself or another user or select one of the existing ones using the Load Criteria option.

   Define the rule's criteria for evaluation using rule syntax, with metadata or data rule. See the following examples:

   • Metadata rule

     For dealing with metadata stored in the database. Syntax is:

     • hasResourceTerm(TermFullyQualifiedName)=1

       Work on the resources that have the given resource term associated with them. For example, hasResourceTerm(Insurance/HomeOwners.State_VA)=1

     • hasFieldTerm(TermFullyQualifiedName)=1

       Work on the fields and columns that have the given field term associated with them. For example, hasFieldTerm(Insurance/HomeOwners.State_VA)=1

   • Data rule

     For dealing with actual data present in the resource. Syntax is to specify the column name directly or use @TermFullyQualifiedName equivalent to the field with the given term name. Use @@customPropertyName to specify the custom property with specific values.

   NoteAll Quality Dimensions are generated only with DataRules.

7. Set the Rule Actions. The rule actions include the following action types.

   You can create rule actions and save them for future use by yourself or another user or select one of the existing ones using the Load Action option.

   You can use the Rule Actions JSON box to view the rule actions as you create them or the rule actions that you have selected using the Load Action option. This helps you visualize the defined rule actions and make the necessary changes. You can also copy the rule actions in the JSON format and use it for REST API requests.

   Action Type	Description
   Add Business Terms	Business Term Select the term name that you want to add. Action Field Specify the field name if the associated action should be performed on a specific field. The field name specified is tagged with the term or terms you selected. If the field name is not specified, then the action will be performed at the resource level. Set Threshold Specify the threshold value at which to perform the rule action.
   Remove Business Terms	Business Term Select the term name that you want to remove. Action Field Specify the field name if the associated action should be performed at the column level. The field name specified is tagged with the term or terms you selected. If the field name is not specified, then the action will be performed at the resource level. Set Threshold Specify the threshold value at which to perform the rule action.
   Set Properties	Select Property Select the property for which you want to set the custom properties. Action Field Specify the field name if the associated action should be performed on a field. Add Property Value Specify the property value. You can specify multiple property values. Each property value should be comma (,) separated with no trailing space. Set Threshold Specify the threshold value at which to perform the rule action.
   Reset Properties	Select Property Select the property for which you want to reset the custom properties. Action Field Specify the field name if the associated action should be performed on a field. Set Threshold Specify the threshold value at which to perform the rule action.
   Set Quality Dimension	Set Quality Dimension (for DataRules) Select the data quality dimension that defines your rule. This dimension is reflected in the data quality graph on the Data Canvas page. Options include: Accuracy The degree to which data correctly describes the "real world" object or event being described. Completeness The proportion of stored data against the business definition of “100% complete”. Consistency The absence of difference when comparing two or more representations of an item against a definition. Each data item is measured against itself or its counterpart in another data set. Note that consistency assessment may not be applicable to all data items. Timeliness The degree to which data represent reality from the required point in time. This is measured by the time distance between each correct and incorrect data point. Uniqueness The inverse of an assessment of the level of duplication. Validity Data is valid if it conforms to the syntax (format, type, range) of its business definition. Typically, this value is the overall measure of data quality. Action Field Specify the field name if the associated action should be performed at the column level. The field name specified is tagged with the term provided in the Set Quality Dimension. If the field name is not specified, then the action will be performed at the resource level Set Low Threshold Specify the threshold value at which to perform the rule action. Set High Threshold Specify the threshold value at which to perform the rule action. You can view the following data quality color range in a donut chart for the column. > High Threshold = green > Low Threshold and < High Threshold = orange < Low Threshold = red For example, If you want to validate the number of records with the state name California. To validate this, you have selected Validity as a quality dimension. Suppose there are a total of 500 records for states; out of these, you want to see records with the state name California. Now, consider that out of 500, we have 50 states with the name California. This is 10% as a validity percentage. Now in the threshold field, if you set the low threshold as 0 (zero) and the high threshold as 9. In this case, 10 is greater than the high threshold, so the data quality chart will show green. If you set a low threshold as 0 (zero) and a high threshold as 15. In this case, 10 falls between 0 and 15, so the data quality chart will show orange. If you set the low threshold as 11 and the high threshold as 15. In this case, 10 falls below the low threshold, so the data quality chart will show red. Similarly, you can set the threshold value for all the data quality dimensions.
   Set Term Assignment State	Set Term Assignment State (for MetaData Rules) Set the assignment state of a term. Select Term Select one or multiple terms for which you want to change the assignment state. If the term is not specified, the selected assignment state is applied to all the terms associated with the resource. Assignment State Select the assignment state. The supported term assignment states are: Accept & Use In Learning Accept & Don’t Use In Learning Reject & Use In Learning Reject & Don’t Use In Learning NoteUse In Learning is applicable at the column level only. At resource level, it is Accept and Reject only.
   Compute Data Quality Score	Compute Data Quality Score Computes the data quality score based on the provided data quality dimension values. Score Formula When you select the Compute Data Quality Score as the rule action type, the template for the score formula is provided in the Score Formula field. You can customize the template formula by changing the dimension weightage values as per the requirement. Score formula template: Accuracy*0.17 + Completeness*0.17 + Consistency*0.16 + Timeliness*0.16 + Uniqueness*0.17 + Validity*0.17 Data quality formula condition: The column level entity should have at least one calculated dimension for this action to execute.. If the rule provides weightages to multiple dimensions, and actual entity has only one dimension, then 0 will be considered as the weightage value for the other dimensions. While defining the formula, there should be at least one valid dimension, and the sum of all dimension weightages should equal to 1. Greater than 1 or less than 1 is not acceptable. Example of a valid formula: Accuracy*0.5 + Completeness*0.20 + Uniqueness*0.30 Action Field Specify the column name if the associated action should be performed only on a specific column, instead of applying the weights on all the columns of the applicable resource. Examples of a valid formula used in the Action Block: Accuracy*0.5 + Completeness*0.20 + Uniqueness*0.30 Accuracy*0.2 + Completeness*0.1 + Consistency*0.1 + Timeliness*0.1 + Uniqueness*0.4 + Validity*0.1
   Set Sensitivity	Set Sensitivity Updates the resource sensitivity level. If multiple Sensitivity rules are executed for a particular resource, then the sensitivity level defined in the latest rule is considered, overriding all the previous rules. Sensitivity Level Specify the sensitivity level. The supported sensitivity levels are: Low Medium High Non Sensitive Unknown Action Item Specify the resource name if the associated action should be performed on a resource. If the resource name is not provided, then the action will change the sensitivity level of all the resources defined in the rule scope and criteria.

8. Click Create Rule.

Parent Topic

• Rule workflow

Procedure

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.

2. Select Business Rules.

   The Business Rules page opens.

3. Locate the business rule you want to configure in the table of rules and select the View Details button (greater-than sign) in its row.

   If you have a large number of rules, select Show Filters to help you find the rule you want to edit.The Business Rule page opens for the selected rule.

4. Edit the fields as needed and click Save Rule.

   The rule is saved with your changes. If there is a problem while creating your rule, an error notification displays at the top of the page. Resolve the error and click Save Rule.

Parent Topic

• Rule workflow

Child Topics

• Update a rule created using rule blocks

Procedure

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.

2. Select Business Rules.

   The Business Rules page opens.

3. Click Management on the menu bar to open the Manage Your Environment page, and then select Business Rules.

   The Business Rules page opens.

4. Use the check box to select the rule you want to delete.

5. Click the Actions menu and then click Remove. Optionally, select the More actions icon, then click Remove from the drop-down menu.

   A message appears prompting you to enter the text Yes to delete the rule.

6. Enter Yes in the input field and click Confirm.

   A message appears confirming that the business rule is deleted.

7. Click Close on the message box to return to the Business Rules page.

Parent Topic

• Rule workflow

Procedure

1. Click Management in the left navigation menu and click Business Rules.

   The list of all rules in Data Catalog is shown.

2. To view the rules associated with the specific resource, click Data Canvas in the left navigation menu > Virtual folders > Specific file.

3. Click Rules tab.

   Rules specific to that file are shown.

Parent Topic

• Rule workflow

Procedure

1. Click Management in the left navigation menu and click Business Rules.

2. Click the Blocks tab.

3. Click Add New Block, and select the scope, criteria, or actions for which you want to create a block.

   NoteIf you make any changes to the block, the rules that are using these blocks are impacted.

Parent Topic

• Rule workflow

-generateReport true -reportName <Name of the report being generated>

Parent Topic

• Execute rules