Searching Data Catalog

Path names searches can match on any part of a path name and are case-sensitive.

Lumada Data Catalog compares the search text to its list of all the path names for all the resources in the catalog. The search is performed as a string comparison where a file, table, or folder is returned when the keyword matches any part of the resources' path.

For example, part matches the file /data/transactions/part-r-00000/data. The keyword actions would match the same file. However, Part would not match the file because while path name searches can match any part of a path containing the search term, they are also case sensitive.

Lumada Data Catalog compares the search text to the names of files, fields, tables, tags, and origins, and the content of tag descriptions and origin descriptions.

When building the keyword search indexes for these items, Data Catalog ensures the metadata values match one or more keywords to any complete token in the index. White space and characters such as single and double quotation marks, question marks, parentheses, carets (^), pound signs (#), colons, periods, hyphens, and commas indicate the end of a word and are otherwise ignored. Words including underscores are not broken across the underscore.

The search is not case sensitive.

For example, risk matches the field name Risk Band. The keyword RISK has the same match behavior as risk, except in path names, which are case-sensitive.

However, risk would not match the field Risk_Band because the token is considered the entire phrase risk_band. To find matches that include the keyword somewhere in the tokenized name, you can use the asterisk wildcard character before, after, or both before and after the keyword. The keyword risk* would match the field Risk_Band.

Keyword searches (other than path names):

• Are case-insensitive
• Match complete words
• Accept the asterisk wildcard character to indicate any preceding or following characters

Characters such as plus sign (+), minus sign (-), ampersand (&), vertical bar (|), exclamation mark (!), carets (^), tilde (~), colon (:), and other special characters are treated as delimiters and are ignored in the search.

If you enter risk-band, the search behaves the same as if you entered risk band.

Lumada Data Catalog crawls the data cluster to discover information from files, Hive tables, and fields inside of files and tables. It groups that information in facets to make it easy for you to search for files or tables with specific characteristics. The facets are categorized as follows:

• File format

  Search for a specific file format.

• Resource type

  Search through a specific resource type.

• Data source

  Search in a specific data source.

• Virtual folder

  Search in a specific virtual folder.

• Processing status

  Search within a specific resource status (like search only profiled resources).

Selecting more than one value inside the same facet includes files that match either value (OR). Selecting more than one value in multiple facets includes files that match both values (AND). If keywords are also specified, the search results match both keywords and facet choices.

By default, Data Catalog provides the following facets:

In addition to using keywords and facets, you can also apply tag-based filters to include or exclude tags and tag children to perform conjunctive and disjunctive searches in an advanced search.

• Including tag(s)

  Enter tag names you want to include in your search. Only selected tag resources and fields are fetched; and, when Include child tags is checked, the children are also returned.

  NoteBusiness entities are blocked from the Include child tags feature. If a business entity tag is selected, search results will not include its children.

• Excluding tag(s)

  Enter tag names you want to exclude in your search. All tag resources and fields are fetched unless excluded; and, when Exclude child tags is checked, all children are returned unless excluded.

For example, when you search for the keyword "Personnel Info" and include the tag US_State, the search results are limited to resources matching the keyword and having the tag (suggested or accepted) US_State. By including or excluding child tags, individual states tagged with US_State can also be filtered.

As with a global search, the available facets visible to the user can further filter out the advanced search results.

You can write expressions to perform queries on searchable property resources or fields using Lumada Data Catalog (LDC) search language. The syntax you use must specify the property or properties to be compared and the operator type wanted for the query. After you have entered the filter string, and following internal validation of the code, the search is executed. Displayed search results depend upon your access control permissions.

To perform a custom search, click in the Search data catalog field then click Go to Advanced Search and select the Custom Search Query tab.

The following table lists the custom query operators in the LDC search language.

You should observe these rules when using Custom Search Query:

• Only searchable properties can be queried. Searches can have a mix of properties. Searches on strings are case sensitive and matches happens on the exact value of the field. Searches on fields can be text_general, and text_with_special_chars, which are case in-sensitive, and matches happen on the terms generated by Solr for the given field value.
• Ranges can be given for numbers and time variables with a combination of greater than and less than operators.
• Multiple statements can be given with a combination of AND and OR operators.
• Statements can be segregated using parenthesis ().
• Provide file sizes in bytes.
• The following date-in-string formats are supported:
  • dd/MM/yyyy hh:mm:ss
  • dd-MM-yyyy hh:mm:ss
  • dd-MM-YYYY
  • dd/MM/YYYY

Syntax examples of custom queries and their meanings are provided below.

If there are more than five facets in any category, you can click View More to display all the facets in a separate dialog box. These are referred to as long facets.

If you select multiple facets in the same category, the resulting filtered search list is an OR filter of the selected facets. If you select multiple facets from different categories, the result is AND filtering.

For example, if you select facets US_State or US_City from the Field Tags category, the resulting list is the OR filtering displaying resources with field tags US_State or US_City. If you also select the facet Accepted from the Field Tag Association State category, the resulting list displays only resources that have field tags US_State or US_City in the Accepted Field Tag Association State.