Managing rules

Last updated
Save as PDF

With Lumada Data Catalog's rules framework you can define, execute, and manage business rules. These rules can evaluate data and metadata properties to add terms, remove terms, and modify custom properties on data assets.

To manage rules, click Management on the menu bar to open the Manage Your Environment page, and then click Business Rules. If you want to create a business rule quickly, click Add Business Rule.

On the Business Rules page, you can author, run, track, and manage all rules in the Data Catalog. You must enter all rules using the Data Catalog rules language. After creating a new rule or updating an existing rule, the rule is available by default.

When you write rules using terms, they are translated automatically into concrete rules that are bound and executed on individual data resources. This translation occurs regardless of which format and platform the resource is in, such as JDBC tables, Hive tables, CSV, Avro, or JSON, as long as it is a format Data Catalog supports. A rule written using terms captures business logic explicitly and can express many concrete rules.

You can view the following sample applications of the rules framework:

Using a rule for sensitive resource tagging
Resource tagging based on data properties

Using a rule for sensitive resource tagging

Many users create business rules to govern sensitive data. The following example identifies and tags all resources containing sensitive data or personal identifiers such as names, addresses, social security numbers, and account information.

Using the Lumada Data Catalog term discovery features, you can identify field metadata and tag data fields such as first name, last name, and address. The Data Catalog built-in terms can identify these fields. Then, you can use a rule to check for any resources that contain tagged sensitive data fields and tag the resources as "PII/Restricted Access".

Although not shown in exact syntax, the following rule example is the only rule you need to write. The rule is term-based and does not depend on an actual field name, resource name, or resource type.

When the rule is processed, it is automatically bound to all qualifying resources and attaches the term you specify when a resource contains the sensitive fields. If you have 100 CSV files, 200 JDBC tables, and 30 Avro files that are all sensitive, they are all labeled correctly after executing this rule.

Metadata rule with field term for sensitive data
Syntax section	Definition
Rule Scope	{ "virtualFolders": [ "DQM" ], "fieldTerms": [ "Built-in_Terms/US_Address", "Built-in_Terms/First_Name", "Built-in_Terms/Last_Name" ], "resourceTerms": [], "sourcePropertyFilters": {}, "termState": [] }
Rule Criteria (rule body)	`hasFieldTerm(Built-in_Terms/First_Name) AND hasFieldTerm(Built-in_Terms/Last_Name) AND hasFieldTerm(Built-in_Terms/US_Address)`
Rule Action	Tag SSN field as "`PII/Restricted Access`"

Resource tagging based on data properties

You can use Data Catalog rules to create a simple data rule that defines a condition and identifies the resources to which the condition applies, and then performs a specified action. In the following example rule, the rule attaches a resource term to all resources where the data for a given field falls within a specified range.

Data rule with resource term
Syntax section	Definition
Rule Scope	{ "virtualFolders": [ "DQM" ], "fieldTerms": [], "resourceTerms": [], "sourcePropertyFilters": {}, "termState": [] }
Rule Criteria (rule body)	`(Category > 100 AND Category < 199) AND TAX_state = 6A`
Rule Action	Tag resource as "`DQM/CA_Employee`"

Rule syntax

A metadata rule executes on all the resources in Data Catalog, applying and evaluating the rule against one resource at a time and executing the specified rule action. However, you can only execute data rules on those resources managed by a specified agent. Additionally, you must use the Data Catalog rules language when entering rules, which provides constructions for expressing scope, conditions, and actions. You can express all these constructions based on actual business terms, field names, custom properties, and business term association states.

Given a rule with Scope S, Criteria B, and Action A, the semantics of the rule can be summarized as "For any resource R that is within S, if B evaluates to true, then perform all actions listed in A on R."

The rule syntax contains three sections:

Rule scope
Sets the scope of resources on which the rule is evaluated and applied.
Rule criteria
Defines the condition in a SQL predicate.
Rule action
Defines the action to take on resources that conform to the rule's evaluation, such as resource tagging, term removal, and setting custom property values.

NoteThe rule syntax requires that you replace the dot in <termDomain>.<Term> with a forward slash. For example, enter Built-in_Terms/Last_Name instead of Built-in_Terms.Last_Name. This replacement applies to all sections of the rule.

Rule scope

The rule scope defines the resources for rule execution. You can define the rule scope by specifying scope types in the Set Rule Scope window.

Virtual folders
You must include at least one virtual folder for the rule to compile. When a single virtual folder is listed, rule execution is run against all the resources in the listed virtual folder. You can enter additional virtual folders as a comma-separated list. If a folder no longer exists when the rule is executed, it is ignored.
Source property filters
A comma-separated list of the source property to filter key-value pairs.
Field terms
A comma-separated list of field-level business terms to further filter the virtual folder resources.
Resource terms
A comma-separated list of resource-level business terms to further filter the virtual folder resources.
Term association states
You can further filter the resources based on the term association state. The possible values are ACCEPTED, REJECTED, and SUGGESTED. If you do not specify a term association state, all states are included.

You can view the different scope types in the following rule scope example.

"ruleScope": {
        "virtualFolders": [
            "DQM"
        ],
        "sourcePropertyFilters": { 
            "domain": "Finance, Banking" 
        },
        "fieldTerms": [
            "Built-in_Terms/Last_Name",
            "Built-in_Terms/US_Address"
        ],
        "resourceTerms": [
            "CA/Employee"
        ],
        "termStates": [
            "ACCEPTED",
            "SUGGESTED"
        ]
    }

Rule criteria

The rule criteria (or rule body) defines the rule that is translated and evaluated into a query for execution against every qualifying resource as defined in the rule scope. You can define the rule body by specifying rule types in the Rule Criteria window.

For example, you can insert a clause that determines what the rule body acts on. The query clause determines if the rule acts on metadata or on actual data from the resource.

Metadata query is compiled using resource metadata
For example, the rule body hasFieldTerm(Built-in_Terms/Social_Security_Number_Delimited) = 1 checks for the presence of the field term Built-in_Terms/Social_Security_Number_Delimited.
Metadata query operating on custom property
For example, the rule body @@business= 'MagnUX' operates on custom properties looking for specific values.
NoteThe inclusion of "@@" indicates the rule is used for a custom properties and is a metadata rule.
Data query is compiled using the resource data
The data query operates on the field terms. For example, the rule body (@EMS/Category >= 100 and @EMS/Category <= 199) and @EMS/Tax_State = '6A' inspects the data in the field tagged with EMS/Category for values between 100 and 199, when the data in the field tagged with EMS/Tax_State has a value of "6A".
NotePrefixing a FieldTerm with an "@" indicates the rule operates on the data tagged by the FieldTerm

Depending on the rule type, the following ruleCriteria queries are possible, where FieldTerm is a full term name including the domain that it is associated with:

Metadata query Data query

Objective: Evaluates against metadata. Rules query for metadata discovered by Data Catalog Objective: Inspect the data when evaluating rules. Rules query for the specific data value identified by the term.

Metadata query	Data query
Objective: Evaluates against metadata. Rules query for metadata discovered by Data Catalog	Objective: Inspect the data when evaluating rules. Rules query for the specific data value identified by the term.
Evaluating on field term `ruleCriteria:` `hasFieldTerm(Domain1/fieldTerm1) AND hasFieldTerm(Domain1/fieldTerm2)`	Evaluating data in fields `ruleCriteria:` `FieldName1 IN (val1,val2, val3) AND FieldName2 = ‘Some Value’`
Evaluating on resource term `ruleCriteria:` `hasResourceTerm(Domain1/ResourceTerm1) OR hasFieldTerm(Domain2/ResourceTerm2)`	Evaluating data in fields with terms `ruleCriteria:` `@Domain1/Term1 = "someValue"` `ruleCriteria:` `@Domain1/fieldTerm1 >= 100 and @Domain1/fieldTerm1 <= 199) and @Domain1/fieldTerm2 = “some_value”`
Evaluating field name `ruleCriteria:` `hasFieldName(fieldname)=1`	Evaluating the length of field values `ruleCriteria:` `length(FieldName1) > 10 OR length(@Domain1/Term1) > 10`
Evaluating custom property `ruleCriteria:` `@@business = 'MagnUX' AND @@strike-count = '3'`	Evaluating the uniqueness of a field `ruleCriteria:` `FieldName1 isunique`
Evaluating nested terms and terms with spaces `ruleCriteria:` hasResourceTerm(Domain1/ParentTerm1.childTerm) OR hasFieldTerm(`Domain name1/field Term2`)	Evaluating the contents of a field `ruleCriteria:` `FieldName1 containsall(‘val1’, ‘val2’, ‘val3’)`
	Evaluating data in fields with nested terms and terms with spaces `ruleCriteria:` @Domain1/ParentTerm1.childTerm >= 100 and @`Domain name1/field Term2` = “value”

Evaluating on field term

ruleCriteria:

hasFieldTerm(Domain1/fieldTerm1) AND hasFieldTerm(Domain1/fieldTerm2)

Evaluating data in fields

ruleCriteria:

FieldName1 IN (val1,val2, val3) AND FieldName2 = ‘Some Value’

Evaluating on resource term

ruleCriteria:

hasResourceTerm(Domain1/ResourceTerm1) OR hasFieldTerm(Domain2/ResourceTerm2)

Evaluating data in fields with terms

ruleCriteria:

@Domain1/Term1 = "someValue"

ruleCriteria:

@Domain1/fieldTerm1 >= 100 and @Domain1/fieldTerm1 <= 199) and @Domain1/fieldTerm2 = “some_value”

Evaluating field name

ruleCriteria:

hasFieldName(fieldname)=1

Evaluating the length of field values

ruleCriteria:

length(FieldName1) > 10 OR length(@Domain1/Term1) > 10

Evaluating custom property

ruleCriteria:

@@business = 'MagnUX' AND @@strike-count = '3'

Evaluating the uniqueness of a field

ruleCriteria:

FieldName1 isunique

Evaluating nested terms and terms with spaces

ruleCriteria:

hasResourceTerm(Domain1/ParentTerm1.childTerm) OR hasFieldTerm(`Domain name1/field Term2`)

Evaluating the contents of a field

ruleCriteria:

FieldName1 containsall(‘val1’, ‘val2’, ‘val3’)

Evaluating data in fields with nested terms and terms with spaces

ruleCriteria:

@Domain1/ParentTerm1.childTerm >= 100 and @`Domain name1/field Term2` = “value”

Rule action

The rule action defines the action to take if the ruleCriteria evaluates to true. A rule action is an array of actions and an action can only apply one term. To apply multiple terms, you must submit a ruleAction for each term.

Actions can be one of the following:

AddBusinessTerms
RemoveBusinessTerms
SetProperties
ResetProperties

When creating a rule action, include the following parts:

actionType
Set to the action taken by the rule.
actionName
Enter the name of the action.
actionAttributes
In the body, use the following guidelines:
- The inclusion of the rule_action_field entry indicates field tagging. The field name specified is tagged with the term provided in the rule_action_term_name.
- The exclusion of the rule_action_field entry implies resource tagging. The resource is tagged with the term provided in the rule_action_term_name.
- The rule_action_threshold entry is used only with a data rule. It defines the percentage of rows that satisfy the rule before the rule action is applied. In a metadata rule, this entry is mandatory and will result in an error if not present. In metadata rules, the recommended threshold value is 0 or 1.

You can define the rule action by specifying action types in the Rule Actions window. The rule action includes the following types:

AddBusinessTerm
When actionType is set to AddBusinessTerm, the ruleAction makes term associations based on rule evaluation. A term suggestion can be applied on a specific field or on a qualifying resource. When applying a term suggestion of a field, the field is identified with a full field name.
NoteThe Data Catalog rule framework does not create new terms. Any term suggestions to be applied as part of rule action must be for existing terms. If an associated term does not exist, Data Catalog displays an error message.
RemoveBusinessTerm
When actionType is set to RemoveBusinessTerm, the ruleAction removes the term associations based on the rule evaluation.
SetProperties and ResetProperties
When actionType is set to SetProperties or ResetProperties, the ruleAction sets or resets custom property values.
Property values are strings. If you specify property names with @@, then the value of its string is substituted for the property name. You can use property actions to set and reset property values. To reset a property value, use ResetProperties.

In the action attributes field, rule_action_property_name is used to mention the custom property name and rule_action_property_value is used to set the value for that property, as in the following example:

"actionAttributes": { 
	"rule_action_property_name": "domain", 
	"rule_action_property_value": "Finance",
	"rule_action_threshold": "1" 
}

The following code sample provides an example of each action type:

[
	{
		"actionType": "AddBusinessTerms",
		"actionName": "",
		"actionAttributes": {
			"rule_action_term_name": "PII/Sensitive",
			"rule_action_threshold": "40",
			"rule_action_field": "SSN"
		}
	},
	{
		"actionType": "AddBusinessTerms",
		"actionName": "",
		"actionAttributes": {
			"rule_action_term_name": "PII/Sensitive",
			"rule_action_threshold": "40"
		}
	},
	{
		"actionType": "AddBusinessTerms",
		"actionName": "",
		"actionAttributes": {
			"rule_action_term_name": "PII/Sensitive",
			"rule_action_threshold": "40",
			"rule_action_stats_field": "SSN"
		}
	},
	{
		"actionType": "RemoveBusinessTerms",
		"actionName": "Remove Field Term",
		"actionAttributes": {
			"rule_action_term_name": "PII/Sensitive",
			"rule_action_threshold": "40",
			"rule_action_field": "SSN"
		}
	},
	{
		"actionType": "RemoveBusinessTerms",
		"actionName": "Remove Resource Term",
		"actionAttributes": {
			"rule_action_term_name": "PII/Sensitive",
			"rule_action_threshold": "40"
		}
	},
	{
		"actionType": "setProperties",
		"actionName": "Update Custom Property value based on threshold",
		"actionAttributes": {
			"rule_action_property_name": "domain",
			"rule_action_property_value": "Finance",
			"rule_action_threshold": "1"
		}
	},
	{
		"actionType": "setProperties",
		"actionName": "Update Custom Property value",
		"actionAttributes": {
			"rule_action_property_name": "domain",
			"rule_action_property_value": "Finance"
		}
	},
	{
		"actionType": "ResetProperties",
		"actionName": "reset proprerty value",
		"actionAttributes": {
			"rule_action_property_name": "domain",
			"rule_action_property_value": ""
		}
	}
]

Requirements for writing rules

You can avoid errors by adopting the following requirements when writing rules:

When using terms in the ruleCriteria for a data query, you must prefix the terms with the @ symbol and then the glossaryname/termname qualifier. In the absence of the @ qualifier, a term glossaryname/termname is interpreted as a column name which may or may not exist and the corresponding results may be misreported.
When evaluating rules to set custom properties, you must prefix the custom property with the @@ qualifier.
Data Catalog supports minimal SQL functions in the rule definition such as AND, OR, <, >, IN, and length().
All terms specified in the actionAttribute field need to pre-exist.
The rule syntax requires that you replace the dot in <termDomain>.<Term> with a forward slash. For example, enter Built-in_Terms/Last_Name instead of Built-in_Terms.Last_Name. This replacement applies to all sections of the rule.
For terms or a glossary with spaces, enclose the term between single quotation marks. For example: @`Glossary name/Term name` > 200
To use field names with spaces, enclose the term between single quotation marks. For example: hasFieldName(`First Name`)=1

Data Catalog's rules framework does not create new terms. It only attaches an existing term to the resource or field specified.

Rule workflow

On the Business Rules page, you can create, update, edit, and delete rules.

Create a rule

Perform the following steps to create a new rule. After you create a rule, you can then configure the rule using Rule syntax.

Procedure

Click Management on the menu bar to open the Manage Your Environment page, and then select Business Rules to open the Business Rules page. Click Add Business Rule.
The Create Business Rule page opens.

Enter the details for your rule:

Field	Description
Business Rule Name (Required)	Enter the unique name of the rule that your users will recognize. Names must start with a letter, and must contain only letters, digits, hyphens, or underscores. White spaces are supported, but trailing spaces are not allowed in names.
Owner	Select the username of the owner of the rule. The default value of this field is the logged-in user.
Description	Enter a description for this rule. For example, you may want to indicate the purpose of the rule to assist other users.
Note	Enter additional comments for the rule. For example, you may want to describe the workflow or use case of the rule.
Rule Enabled	By default, a new rule is enabled. When you select Rule Enabled, all referenced names (such as custom properties, terms, fields, resources, and virtual folders) are verified for accuracy in the system. Clear the check box to disable the rule. When a Rules Execution job is run, disabled rules are skipped and are not evaluated.

Click Create Business Rule to save your rule.
The rule is created and the details are saved.

Next steps

Configure a rule.

Configure a rule

After you create a business rule, you can configure it in the Configuration view of the Business Rule page. This task assumes you have completed Create a rule and are on the Business Rules page.

Procedure

If you have not already done so, locate the business rule you want to configure in the table of rules and select the View Details button (greater-than sign) in its row.
The Business Rule page opens to the Details view for the selected rule.
Click the Configuration tab.

Enter the following configuration values for the rule:

Field	Description
Data Quality Dimensions	Select the data quality dimension that defines your rule. This dimension is reflected in the data quality graph on the Data Canvas page. Options include: None This rule is not used as a data quality metric. Completeness The proportion of stored data against the business definition of “100% complete”. Consistency The absence of difference when comparing two or more representations of an item against a definition. Each data item is measured against itself or its counterpart in another data set. Note that consistency assessment may not be applicable to all data items. Uniqueness The inverse of an assessment of the level of duplication. Validity Data is valid if it conforms to the syntax (format, type, range) of its business definition. Typically, this value is the overall measure of data quality.
Execute Rule	By default, the rule is set to run manually.
Advanced Mode	Select this check box to enter parameters for the rule. Clear this check box if you do not want to enter additional parameters.
Set Rule Scope	Define the filters for evaluating the rule field. Rule scope includes the following parameters: Virtual Folders Specify comma-separated virtual folders on which the rule should be executed. ResourceTerms Specify comma-separated business terms to filter the virtual folder resources. FieldTerms Specify comma-separated field terms to filter the resource terms. sourcePropertyFilters Specify property name and value to filter. termState Specify the state of the business term associations. Can be ACCEPTED, REJECTED, or SUGGESTED. If nothing is specified, it includes resources with any state.
Rule Criteria	Define the rule's criteria for evaluation using rule syntax, with MetadataRule or DataRule. See the following examples: MetadataRule For dealing with metadata stored in the database. Syntax is: `hasResourceTerm(TermFullyQualifiedName) =1` Work on the resources that have the given resource term associated with them `hasFieldTerm(TermFullyQualifiedName) =1` Work on the fields/columns that have the given field term associated with them DataRules For dealing with actual data present in the resource. Syntax is to specify the column name directly or use `@TermFullyQualifiedName` equivalent to the field with the given term name. Use `@@customPropertyName` to specify the custom property with specific values.
Rule Actions	Define the actions taken after the rule evaluation is accepted as true. This action can be associating a field or resource term, removing an associated term, or updating a custom property value. The `rule_action_stats_field` represents the field on which the data quality metric result is to be displayed. Specify the appropriate parameters for the following actions: `AddBusinessTerms` or `RemoveBusinessTerms` `rule_action_term_name`: term name that should be added or removed `rule_action_threshold`: threshold value at which to perform the rule action `rule_action_field_name`: specify the field name if the associated action should be performed on a field `SetProperties` or `ResetProperties` `rule_action_property_name`: custom property name `rule_action_property_value`: value of the custom property

If the rule configuration values are entered correctly, click Save Changes.
If there is a problem while saving your rule, an error message appears indicating the problem. Fix the problem and save your changes.
To execute the rule, click Run Now next to the Execute Rule field.
A confirmation message appears indicating that the business rule is submitted to jobs management and a notification is created for the user. Optionally, click View Details in the notification to check the status of the rule submission job.

Results

Your created and configured business rule appears on the Business Rules page. Select the rule if you want to run, edit, or remove the rule.

Update a rule

If you have already created and configured a rule, you can edit it from the Business Rules page.

Perform the following steps to edit a rule:

Procedure

Click Management on the menu bar to open the Manage Your Environment page, and then select Business Rules to open the Business Rules page.
The Business Rules page opens.
Locate the business rule you want to configure in the table of rules and select the View Details button (greater-than sign) in its row.
If you have a large number of rules, select Show Filters to help you find the rule you want to edit.The Business Rule page opens to the Details view for the selected rule.
Edit the fields as needed in the Details and Configuration views.
Click Save Changes.
The rule is saved with your changes. If there is a problem while creating your rule, an error notification displays at the top of the page. Resolve the error and click Save Changes.

Delete a rule

If a rule is no longer needed, you can delete it. Perform the following steps to delete a rule:

Procedure

Click Management on the menu bar to open the Manage Your Environment page, and then select Business Rules.
The Business Rules page opens.
Use the check box to select the rule you want to delete.
Click the Actions menu and then click Remove. Optionally, select the More actions icon, then click Remove from the drop-down menu.
A message appears confirming your business rule is now deleted.
Click Close on the message box to return to the Business Rules page.

Execute rules

Perform the following steps to execute a Data Catalog rule.

Go to Management and click Business Rules.
Select the check box next to the rule you want to run, and click the Actions button.
Select Run Now from the Actions menu.
(Optional) If you want to enter parameters, select the Advanced Mode check box, and enter the parameters in the text box.
Click Run Now.

Rule execution report

A rule execution report is a report of all the rules that summarizes how well a rule evaluates the resources in Data Catalog.

To generate a rule execution report, enter the following additional parameters before rule execution:

-generateReport true -reportName <Name of the report being generated>

Define the options as follows:

-generateReport
If this parameter is passed, rule execution generates a report with the name specified by the -reportName parameter.
-reportName
User-defined name for the report being generated. Use with the -generateReport parameter.

All reports are generated in the /var/log/ldc/generatedReports directory. If you do not provide a report name, Data Catalog randomly generates a unique name for each report.

Sample rules

You can use these examples of metadata and data rules to help you write rules for your implementation of Data Catalog:

Metadata rule samples
Data rule samples

Metadata rule samples

A metadata rule works with the metadata associated with the resource or field. The included examples of metadata rules illustrate the following scenarios:

Field term binding using a field name
Field term binding using a match of a field name
Custom property setting and field term removal

Field term binding using a field name

The following sample rule validates the presence of Built-in_Terms/Social_Security_Number_Delimited for all resources containing the field terms Built-in_Terms/Last_Name and Built-in_Terms/Address either in ACCEPTED or SUGGESTED state within the DQM virtual folder, then applies the resource term PII/Sensitive to the field named "SSN".

"name" : "Sensitive Term Completeness",
"description" : "If field terms Built-in_Terms/US_Address, Built-in_Terms/Last_Name and Built-in_Terms/Social_Security_Number_Numeric are present then add PII/Sensitive to SSN field"
"ruleScope" : 	{
			"virtualFolders": [
				"DQM"
			],
			"fieldTerms": [
				"Built-in_Terms/US_Address",
				"Built-in_Terms/Last_Name"
			],
			"resourceTerms": [
				"CA/Employee"
			],
			"sourcePropertyFilters": {
				"domain": "Finance, Banking"
			},
			"termState": [
				"ACCEPTED",
				"SUGGESTED"
			]
		}

"ruleCriteria" : "hasFieldTerm(Built-in_Terms/Social_Security_Number_Numeric)=1"
 
"ruleActions" : [
			{
				"actionType": "AddBusinessTerms",
				"actionName": "",
				"actionAttributes": {
					"rule_action_term_name": "PII/Sensitive",
					"rule_action_threshold": "1",
					"rule_action_field": "SSN"
				}
			}
		]

ruleCriteria
This field validates the presence of field term Built-in_Terms/Social_Security_Number_Delimited in the resources.
ruleScope
This field applies various filters or scopes the rule to specific resources:
- virtualFolders
  This field limits the rule evaluation to only the virtual folder named DQM.
- fieldTags
  This attribute further filters resources that contain the field terms Built-in_Terms/Last_Name and Built-in_Terms/US_Address.
- resourceTerms
  This field further limits evaluation to resources containing the CA/Employee resource term.
- termState
  This field restricts the rule evaluation rule to consider terms in either the ACCEPTED or SUGGESTED states.
ruleAction
This field specifies the attributes to apply the action, in this case to attach PII/Sensitive to the field SSN in all resources the which the rule applies.
rule_action_threshold
This field is mandatory but not used for metadata rules. It should be set to 0 or 1. If it is not present, an error will occur.

Field term binding using a match of a field name

The following sample rule validates the presence of the field term Built-in_Terms/Email in all resources within the DQM, Holdings, and Bank_Retail virtual folders, then applies the field term PII/contactType to the fields 'email'.

"ruleScope" : {	
	"virtualFolders": [
	"DQM, Holdings, and Bank_Retail"
	],
	"fieldTerms": [
	"Built-in_Terms/Email"
	],
	"resourceTerms": [],
	"sourcePropertyFilters": {},
	"termState": [
	"ACCEPTED",
	"SUGGESTED"
	]
}

"ruleCriteria" : "hasFieldName(email)=1"
 
"ruleActions" : [
	{
	"actionType": "AddBusinessTerms",
	"actionName": "",
	"actionAttributes": {
	"rule_action_term_name": "PII/contactType",	
	"rule_action_threshold": "1",
	"rule_action_field": "email"
	}
}

ruleCriteria
This field validates the presence of the field name 'email' in the resources.
ruleScope
This field applies various filters or scopes the rule to specific resources:
- virtualFolders
  This field limits the rule evaluation to only the virtual folders named DQM, Holdings and Bank_Retail.
- termState
  This field restricts the rule evaluation to terms in either the ACCEPTED or SUGGESTED states.
- fieldTerms
  This field limits the rule evaluation to only fields with the term Built-in_Terms/Email.
ruleAction
This field specifies the attributes to apply the action, in this case to attach PII/contactType as a field term to all fields containing email in all qualifying resources.
rule_action_threshold
This field specifies the minimum percentage evaluation match for the action. In the above example, all the resources where 0% or more of the data passes the rule evaluation are tagged with the PII/contactType field term.

Custom property setting and field term removal

In the following example, the rule is updating a custom property value and removing a field term. The rule checks for all resources with the resource term DQM/Employee and field term DQM/CA_Tax.

It checks if the resource is tagged as DQM/CA_Tax and also validates the value of the custom property data_owner, checking to see if the value is "Lara". If both of these conditions are satisfied, the rule updates the value of data_owner to "Joe" and removes the term DQM/CA_Tax.

"ruleScope" :   {
            "virtualFolders": [
                "DQM"
            ],
            "fieldTerms": [],
            "resourceTerms": [
                "DQM/Employee",
                "DQM/CA_Tax"            
            ],
            "sourcePropertyFilters": {
                "data_owner" : "Lara"
            },
            "termState": [
                "ACCEPTED",
                "SUGGESTED"
            ]
        }
"ruleCriteria" : "@@data_owner="LARA" AND hasResourceTerm(DQM/CA_Tax)=1"
"ruleActions" : [
            {
                "actionType": "RemoveBusinessTerms",
                "actionName": "Remove Field Term",
                "actionAttributes": {
                    "rule_action_term_name": "DQM/CA_Tax",
                    "rule_action_threshold": "1"
                }
            },
            {
                "actionType": "setProperties",
                "actionName": "Update Custom Property value",
                "actionAttributes": {
                    "rule_action_property_name": "data_owner",
                    "rule_action_property_value": "Joe"
                }
            }
        ]

ruleScope
This field scopes the rule to the following filters:
- resourceTerm
  This field filters the resources with the DQM/Employee resource term and the DQM/CA_Tax resource term.
- virtualFolders
  This field limits the rule evaluation to only the virtual folders named DQM.
- termState
  This field restricts the rule evaluation rule to consider terms in either the ACCEPTED or SUGGESTED states.
ruleCriteria
This field inspects and validates the data values for a field tagged with DQM/CA_Tax and that the value of the custom property data_owner is "Lara".
ruleActions
This rule element contains an array of two actions:
- Updating custom property value
  - rule_action_property_name
    This field specifies the property name for which the value is updated. In this case, the property name is data_owner.
  - rule_action_property_value
    This field specifies the new value to which the custom property is updated. In this case, the value is "Joe".
  - rule_action_threshold
    This field is intentionally left out for rule actions on custom properties.
- Remove field action
  - rule_action_term_name
    This field specifies the term DQM/CA_Tax to be removed.
  - rule_action_field
    This field is intentionally left out for the resource term.
  - rule_action_threshold
    This field specifies the minimum percentage evaluation match for the action to apply to the resource. For metadata rules, the threshold value is not used, but it is generally set to 0 or 1.

Data rule samples

A data rule inspects the data for a field or field term when evaluating a resource. The following example of a data rule illustrates resource term and field term binding.

Resource term and field term binding

This rule is for resource term binding on resources that contain a specified type of data. It checks for all resources with the resource term DQM/Employee and examines the data in fields tagged with DQM/taxCode and DQM/stateCode for qualifying data, then attaches the resource term DQM/CA_Employee and DQM/CA_Tax term to the field taxcode.

"name" : "CA_Term Rule",
"ruleScope" : 	{
			"virtualFolders": [
				"DQM"
			],
			"fieldTerms": [
				"DQM/taxCode",
				"DQM/stateCode"
			],
			"resourceTerms": [
				"DQM/Employee"			
			],
			"sourcePropertyFilters": {},
			"termState": [
				"ACCEPTED",
				"SUGGESTED"
			]
		}

"ruleCriteria" : "(@DQM/taxCode >= 100 and @DQM/taxCode <= 199) and @DQM/stateCode = '6A'"
 
"ruleActions" : [
			{
				"actionType": "AddBusinessTerms",
				"actionName": "CA ResourceTerm Tagging",
				"actionAttributes": {
					"rule_action_term_name": "DQM/CA_Employee",
					"rule_action_threshold": "50"
				}
			},
			{
				"actionType": "AddBusinessTerms",
				"actionName": "CA Field Term Tagging",
				"actionAttributes": {
					"rule_action_term_name": "DQM/CA_Tax",
					"rule_action_threshold": "50",
					"rule_action_field": "taxcode"
				}
			}
		]

The rule elements are applied as follows:

ruleScope
The ruleScope element scopes the rule to the following filters:
- virtualFolders
  This field limits the rule evaluation to only the virtual folders named DQM.
- fieldTerms
  This field restricts the rule evaluation to only the fields DQM/taxCode and DQM/stateCode.
- resourceTerms
  This field filters the resources that have the DQM/Employee resource term.
- termState
  This field restricts the rule to terms in either the ACCEPTED or SUGGESTED states.

ruleCriteria
The ruleCriteria element limits the data to resources with fields tagged with DQM/taxCode that have values between 100 and 199, and the data in the field tagged with DQM/stateCode contains the value "6A".
ruleActions
The ruleActions element is an array of two actions:
- Resource tagging action
  - rule_action_term_name
    This field specifies the new term DQM/CA_Employee to bind to qualifying resources.
  - rule_action_field
    This field is left out intentionally to indicate resource tagging.
  - rule_action_threshold
    This field specifies the minimum percentage evaluation match for the action to apply. In the above example, all the resources where 50% or more of the data passes the rule evaluation are tagged with the DQM/CA_Employee resource term.
- Field tagging action
  - rule_action_term_name
    The rule_action_term_name from one action specifies the new term DQM/CA_Employee to bind to qualifying resources, while the other action specifies the DQM/CA_Tax term that binds to a field specified in the rule_action_field.
  - rule_action_field
    The presence of rule_action_field indicates field tagging, and the reference field taxcode is used for binding the DQM/CA_Tax term.
  - rule_action_threshold
    The rule_action_threshold field specifies the minimum percentage evaluation match for the action to apply. For example, all the resources where 50% or more of the data passes the rule evaluation are tagged with the DQM/CA_Employee resource term.

Field term binding after checking field length and set of values

This rule is to check the length of the field with the term DQM/Tax Code and to examine the data in fields with term DQM/Employee Dept with a set of values. If the condition is greater than the threshold value then add a term DQM/CA Employee to the resource and display the DQ metrics on the field EmployeeId.

Rule Scope: 
{
	"virtualFolders": [
		"DQM"
	],
	"fieldTerms": [
		"DQM/Employee Dept",
		"DQM/Tax Code"
	],
	"resourceTerms": [],
	"sourcePropertyFilters": {},
	"termState": []
}
 
Rule Criteria:
length(@`DQM/Tax Code`) < 8 AND @`DQM/Employee Dept` IN ("Finance", "Banking", "Marketing", "Sales")
 
Rule Actions:
[
	{
		"actionType": "AddBusinessTerms",
		"actionName": "",
		"actionAttributes": {
			"rule_action_term_name": "DQM/CA Employee",
			"rule_action_threshold": "70",
			"rule_action_stats_field": "EmployeeId"
		}
	}
]

The rule elements are used as follows:

ruleScope

This field scopes the rule to the following filters:
- virtualFolders
  This field limits the rule evaluation to only the virtual folders named DQM.
- fieldTerms
  This field filters the resources with the DQM/Employee Dept and DQM/Tax Code field terms.
- termState
  This field restricts the rule evaluation rule to consider terms with the mentioned status. If no value is preset, all statuses are considered for evaluation.
ruleCriteria
This field inspects and validates the length of field with the term DQM/Tax Code and the data in the field with the term DQM/Employee Dept against the list of values mentioned.
ruleActions
Uses the following filters to add a resource term action:
- rule_action_term_name
  This field specifies the term DQM/CA Employee to be added.
- rule_action_threshold
  This field specifies the minimum percentage evaluation match for the action to apply to the resource. In this case, if the result is more than 70%, then the rule action is executed.
- rule_action_stats_field
  This field indicates on which field the DQ metrics must be mentioned.

Pentaho+ documentation has moved!

The new product documentation portal is here. Check it out now at docs.hitachivantara.com.

Using a rule for sensitive resource tagging

Resource tagging based on data properties

Rule syntax

Rule scope

Rule criteria

Rule action

Requirements for writing rules

Rule workflow

Create a rule

Configure a rule

Update a rule

Delete a rule

Execute rules

Rule execution report

Sample rules

Metadata rule samples

Data rule samples