Skip to main content

Pentaho+ documentation is moving!

The new product documentation portal is here. Check it out now at docs.hitachivantara.com

 

Hitachi Vantara Lumada and Pentaho Documentation

Manage virtual folders

Parent article

You can use virtual folders in Lumada Data Catalog to create groups of resources belonging to a data source for easier management. Additionally, data resources can be added to multiple virtual folders. When used correctly, virtual folders are a key part in controlling the accessibility and visibility of the data.

When you create a data source in Data Catalog, you also create a corresponding top level root virtual folder. This virtual folder includes all the resources that are part of the data source connection that you created or added when creating a data source. The root virtual folder acts as a building block from which more virtual folders can be created. Additionally, data resources can be added to multiple virtual folders.

To manage virtual folders, click Management in the left navigation menu and then click Virtual Folders.

Repository-only browsing

All browsing of data assets is done on metadata in the repository. Users are not required to enter their own credentials to access any data.

Add a virtual folder

If you have administrative privileges in Data Catalog, you can create more virtual folders based on the root virtual folders or other non-root virtual folders.
NoteIn the Virtual Folder page, the root virtual folder is denoted by a dot (.) inside the folder icon, as shown here: Root folder icon

Perform the following steps to add a virtual folder:

Procedure

  1. Click Management in the left navigation menu.

    The Manage Your Environment page opens.
  2. Click Virtual Folders to open the Virtual Folders page, and then click Add Virtual Folder.

    Optionally, select Add Virtual Folder directly from the Manage Your Environment page.The Create Virtual Folder page opens.
  3. Enter the following details in the Create Virtual Folder page.

    FieldDescription
    Folder Name (Required)The name of the virtual folder. The virtual folder name must begin with an alphabetic character and can include alphanumeric characters, hyphens, and underscores. It cannot contain dots (.) because Data Catalog uses the dot (.) separator to identify lineage or parentage between nested folders.
    DescriptionA description to help identify the audience for this folder. This description is just to assist the administrator. Data Catalog does not use this field for metadata processing on that folder.
    Parent Virtual FolderThe parent root virtual folder or another virtual folder from which the new virtual folder is derived. Start typing the name of the database in the field and select the best match from the list that displays. Only folders accessible to your role are listed.
    NoteYou must specify an existing virtual folder as a parent for your new virtual folder.
    Expert UsersSelect one or more expert users who are skilled at using the resource. Expert users must first be defined as users in Lumada Data Catalog.
    NoteExperts can only be assigned to non-root virtual folders.
    Source Path (Required)The absolute path to the resource folder included in this virtual folder. The resource folder should be a subset of the parent folder.
    Include Pattern (Optional)A regular expression you want included when filtering the source. See Using regular expressions.
    Exclude Pattern (Optional)A regular expression you want excluded when filtering the source. See Using regular expressions.
  4. Click Validate Virtual Folder.

    Data Catalog verifies the specified Source Path along with the resulting list included in the newly created virtual folder. The validation is based on the include and exclude regular expressions you define.
    NoteTo successfully validate regular expression patterns, make sure the Force iterate through all tables of a schema/database discovery configuration setting is set to true.
  5. Click Create Virtual Folder.

    If this button is unavailable, go back to the previous step or troubleshoot your validation of the virtual folder.

Results

Your virtual folder is now ready for browsing and processing.

Using regular expressions

When working with virtual folders, you can specify regular expressions for both paths and include and exclude patterns, which gives you more granular access control for the data source while maintaining the privacy of sensitive data.

ImportantData Catalog uses the JDK regular expression (regex) engine to process regular expressions. Make sure your regular expressions conform to JDK regex syntax so Data Catalog can process them accurately.

By default, everything in the Source Path is included, and nothing is excluded. The default Include Pattern is ".*" and the default Exclude Pattern is EMPTY.

The string you specify for the full source path of a virtual folder equals the source path specification string plus the Include Pattern regular expression string minus the Exclude Pattern regular expression string. For example, as the administrator, you may decide to restrict access. In the Create Virtual Folder page, you give a R_DataQM sample virtual folder access to all CSV files in the /user/ldcsvc/DQM path beginning with the R.* regular expression.

The / is prepended to the Include Pattern and Exclude Pattern regular expressions in this example. Without the /, the virtual folder validation will return zero results. In some cases, you may need to escape the / by adding a backslash \. You can include or exclude more than one resource by listing them as a pipe | separated list within (...) brackets, for example, (/user/ldcsvc/Pass1/.* | /user/ldcsvc/Pass2/.*)

You can use ".*" to match any character. For example, test1.emp.* matches paths such as test1.emp1 and test1.emp2. You can escape "." by using a backslash (\) in front of it. For example, /warehouse180_patch2.*|/warehouse104_patch1.*.

Regular expression examples of virtual folder definitions

The following table lists samples of virtual folder definitions for a data source named MyHive and the source path (root)/. Use the samples to help create your own regular expresssions of virtual folder definitons.

SampleDefinition
All schemas starting with az_ and all the tablesFor this sample, use the following settings:
  • Source Path: /
  • Include Pattern: "az_.*"
  • Exclude Pattern: ""
Tables starting with emp or dept in the schema called az_dbFor this sample, use the following settings:
  • Source Path: /az_db
  • Include Pattern: "emp.*|dept.*"
  • Exclude Pattern: ""
All tables in the schema called az_db, except tables ending with _viewFor this sample, use the following settings:
  • Source Path: /az_db
  • Include Pattern: ".*"
  • Exclude Pattern: ".*_view"
All tables in schemas starting with test1 or test2, except tables ending with _viewFor this sample, use the following settings:
  • Source Path: /
  • Include Pattern: "(test1.*|test2.*)"
  • Exclude Pattern: ".*_view"
All tables in schemas test1 or test2 with names starting with emp.For this sample, use the following settings:
  • Source Path: /
  • Include Pattern: "(test1.emp.*|test2.emp.*)"
  • Exclude Pattern: ""

Update a virtual folder

For root virtual folders, you can only update the Description and Expert Users fields. You can also update the Folder Name field of non-root or derived virtual folders. Root virtual folders assume the same name as their corresponding data source connection when the data source was first added.
NoteTo change any other field for the virtual folder, such as Source Path or Include Pattern or Exclude Pattern, you must delete the current virtual folder and create a new one.

Perform the following steps to update a virtual folder:

Procedure

  1. Click Management in the left nvigation menu to open the Manage Your Environment page, then click Virtual Folders.

    The Virtual Folders page opens.
  2. Locate the virtual folder you want to update in the table of folder names and click the View Details button (greater-than sign) in its row.

    If you have a large number of folders, click Show Filters on the Virtual Folders page to help you find the folder you want to edit.The VFolder details page opens for the selected virtual folder.
  3. If you are updating non-root virtual folder, modify the Folder Name field, as needed.

  4. Edit the following fields as needed:

    FieldDescription
    DescriptionA description to help identify the audience for this folder. This description is just to assist the administrator. Data Catalog does not use this field for metadata processing on that folder.
    Expert UsersExpert users are typically proficient at using the resource. To be assigned as experts, these users first need to be defined in Lumada Data Catalog.
  5. Select Validate Virtual Folder. Data Catalog verifies the specified Source Path along with the resulting list included in the newly updated virtual folder. The validation is based on the include and exclude regular expressions you define.

  6. Click Save Changes to keep your changes.

    If this button is unavailable, go back to the previous step or troubleshoot your validation of the virtual folder.

Remove a virtual folder

You can remove a non-root virtual folder in one of two ways, from the Virtual Folders page or from the VFolder details page. Only an administrator can remove a root virtual folder by deleting its corresponding data source connection.
AttentionWhen you remove a parent virtual folder, any nested or child virtual folders dependent on the parent are also removed.

Perform the following steps to remove a non-root virtual folder:

Procedure

  1. Click Management in the left navigation menu to open the Manage Your Environment page, then click Virtual folders.

    The Virtual Folders page opens.
  2. Locate the virtual folder you want to remove in the table of folder names.

    If you have a large number of folders, click Show Filters to help you find the folder you want to remove.
  3. Perform one of the following actions to delete a virtual folder:

    • Click the More Actions () icon at the end of the row for the selected folder, then click Remove.
    • Click the View Details button (greater-than sign) in the row to open the VFolder details page, and then click Remove Virtual Folder.
    A confirmation dialog box appears.
  4. Click Remove Virtual Folder in the confirmation dialog box to proceed.

Results

The virtual folder is removed.