Skip to main content

Pentaho+ documentation has moved!

The new product documentation portal is here. Check it out now at docs.hitachivantara.com

 

Hitachi Vantara Lumada and Pentaho Documentation

Manage virtual folders

Parent article

Using virtual folders in Lumada Data Catalog, you can create groups of resources belonging to a data source for easier management. In addition, you can have data resources in multiple virtual folders. When you create a data source, its corresponding top level virtual folder, referred to as the root virtual folder, is also created. This virtual folder includes all the resources that are part of the data source connection that you created or added when creating or adding a data source. The root virtual folder acts as a building block from which more virtual folders can be created.

When used correctly, virtual folders can also play a key part in access control implementations by controlling the access to data or visibility of data.

To manage virtual folders, navigate to Manage and then click Virtual folders.

Browse virtual folders in the repository

Repository-only browsing

You can use repository-only browsing to browse data resources that have already been created in the Lumada Data Catalog repository making browsing more responsive because you do not need to create connections to remote data sources.

Because repository-only browsing along with caching may provide stale information about the data resources in a data sources, click the Refresh button on the Virtual Folders page to refresh the list of data resources from a source when new resources have been added to the data source.

Create a virtual folder

In Lumada Data Catalog, you can create more virtual folders based on the root virtual folders or other non-root virtual folders.

Follow the steps below to create a virtual folder:

Procedure

  1. Navigate to Manage and then click Virtual folders.

  2. Click Create a Virtual Folder.

    Browsing to Virtual Folders The Create Virtual Folder dialog box opens.
  3. In the Create Virtual Folder dialog box, enter the following details.

    Create Virtual Folder dialog box
    1. In the Folder Name field, enter a name for the virtual folder.

      The virtual folder name must begin with an alphabetic character and can include alphanumeric characters, hyphens, and underscores. It cannot contain dots . because Data Catalog uses the '. separator to identify lineage or parentage between nested folders.
    2. In the Folder Description field, enter a description that can help identify the audience for this folder.

      This description is just to assist the administrator. Data Catalog does not use this field for metadata processing on that folder.
    3. In the Parent Folder field, specify the parent root virtual folder or another virtual folder from which the new virtual folder is derived. Start typing the name of the database in the field and select the best match from the list that displays. Only folders accessible to your role will be listed.

      NoteTo create a new virtual folder, you must specify an existing virtual folder as a parent.
    4. Specify the Experts for this virtual folder.

      Experts are typically users that are proficient using the resource. To be assigned as experts, these users first need to be defined in Lumada Data Catalog.

      NoteExperts can only be assigned to non-root virtual folders.
    5. Click + Add path specification, then enter the Source Path.

      Specify the absolute path to the resource folder that will be included in this virtual folder. Note that this resource folder should be a subset of the parent folder.
    6. (Optional) Specify regular expressions using Include Pattern and Exclude Pattern. See Using regular expressions.

    7. Click Validate Virtual Folder.

      Data Catalog validates the specified source path along with the resulting list included in the newly created virtual folder. The validation is based on the include and exclude regular expressions that have been defined.

      NoteTo successfully validate regular expression patterns, enable the ldc.discovery.selector.forceIterate property in <LDC Install Dir>/app-server/conf/configuration.json.
      Example Validation details
  4. Click Create to finish creating a virtual folder.

    This virtual folder is now ready for browsing and processing.
    NoteIn the Browse list view, the root virtual folder is denoted by a dot . on the lower right side of the icon.

Using regular expressions

When working with virtual folders, you can specify regular expressions for both paths and include and exclude patterns, which gives you more granular access control for the data source while maintaining the privacy of sensitive data.

By default, everything under Source Path is included and nothing is excluded. Therefore, the default Include Pattern is ".*" and the default Exclude Pattern is EMPTY.

The full path string of a virtual folder equals the source path specification string plus the Include regular expression string minus the Exclude regular expression string. For example, the administrator may decide to restrict access. In the Create Virtual Folder dialog box, the administrator gives the R_DataQM virtual folder access to all csv files in the /user/waterlinesvc/DQM path beginning with the R.*. Note that the / is prepended to the Include and Exclude regular expressions in this example. Without the /, the virtual folder validation will return 0 results. In some cases, you may need to escape the / by adding a backslash \.

You can include or exclude more than one resource by listing them as a pipe | separated list within (...) brackets, for example, (/user/waterlinesvc/Pass1/.* | /user/waterlinesvc/Pass2/.*)

Example virtual folder definitions for a data source MyHive with path (root) /
  • All schemas starting with "az_" and all the tables.

    • Source Path: /
    • Include pattern: "az_.*"
    • Exclude pattern: ""
  • Tables starting with emp or dept in the schema called az_db.

    • Source Path: /az_db
    • Include pattern: "emp.*|dept.*"
    • Exclude pattern: ""
  • All tables in the schema called az_db, except those ending with _view.

    • Source Path: /az_db
    • Include pattern: ".*"
    • Exclude pattern: ".*_view"
  • All tables in schemas starting with test1 or test2, except tables ending with _view.

    • Source Path: /
    • Include pattern: "(test1.*|test2.*)"
    • Exclude pattern: ".*_view"
  • All tables in schemas test1 or test2 with names starting with emp.

    • Source Path: /
    • Include pattern: "(test1.emp.*|test2.emp.*)"
    • Exclude pattern: ""
    • ".*" is used to match any character.
    • "." can be escaped by using a "\" in front. So, test1.emp.* matches paths such as test1.emp1 and test1.emp2
    • /warehouse180_patch2.*|/warehouse104_patch1.*

Update a virtual folder

The only permitted updates on root virtual folders are the Description, Experts, and System name fields. For non-root or derived virtual folders, the Name of the derived (non-root) virtual folder can be updated as well. To change any other attribute of the virtual folder, such as path or include or exclude patterns, you must delete the current virtual folder and create a new one.

Root virtual folders assume the same name as their corresponding data source connection when the data source was first added.

Virtual folder types

Follow the steps below to update a virtual folder:

Procedure

  1. Navigate to Manage, click Virtual folders, and then select the desired virtual folder.

    If the virtual folder already has a description, it is below the folder name and is clickable. If no description exists, + Add a description appears under the folder name.
  2. Click the existing description at the top of the page to change it, or click + Add a description to enter edit mode and add the description.

  3. Click the check mark to save your changes.

  4. If the folder is a non-root virtual folder, then in the Name field, modify the folder name.

  5. Click System name to add or change the system name.

    AttentionSystem names must be unique. You can only set system names after the virtual folder has been created as part of an update.
  6. If the folder is a non-root virtual folder, in the EXPERT USERS field, add or remove experts.

  7. Click Validate Virtual Folder to check the specified source path along with the resulting list included in the updated derived virtual folder based on the defined include and exclude regular expressions. You can also select the Skip Validate Virtual Folder check box to skip validation.

    The resulting list included in the updated derived virtual folder is not applicable for root virtual folders.
  8. Click Save to keep your changes.

Delete a virtual folder

You can delete a virtual folder in one of two ways. Both methods are described below.

AttentionWhen you delete a parent virtual folder, any nested or child virtual folders that are dependent on this parent are also deleted. In addition, root virtual folders can only be deleted by the administrator by deleting the corresponding data source connection.

Procedure

  1. Perform one of the following actions to delete a virtual folder:

    • In the list view for virtual folders, hover over the desired folder until the More actions icon appears and then select Delete.
    • In the Settings menu for the virtual folder, click the More actions icon and then select Delete.
    Deleting a virtual folder A confirmation dialog box displays.
  2. Click Delete to proceed.

    Delete virtual folder confirmation

Results

The virtual folder is deleted.