Installing Data Catalog on Kubernetes with Helm

Last updated
Save as PDF

Use this installation method if all your data sources are in networks separated by low latencies from the Kubernetes cluster and if you do not have any Hadoop-based data sources (HDFS or Hive). In this installation, all the necessary Data Catalog components are entirely deployed in the Kubernetes cluster.

Before you begin

Before you begin to install Data Catalog on Kubernetes, you must obtain the relevant Helm chart and Docker images.

Contact Hitachi Vantara Lumada Data Catalog support to obtain access to the following artifacts:

Helm chart
ldc-7.0.0.tgz
Docker images

By default, the Hitachi Vantara-owned Docker images are not publicly available. The following Docker images files will be provided by Hitachi Vantara:
- lumada-catalog/app-server:7.0.0
- lumada-catalog/agent:7.0.0
- lumada-catalog/mongodb-migration-tool:7.0.0
- lumada-catalog/spark:3.1.1-hadoop-2.10.1-1.0.0
- lumada-catalog/mongodb:5.0.6-220222.21-master-ee

Installing on Kubernetes

Perform the following tasks to install Data Catalog on Kubernetes with Helm:

Load Docker images and Helm charts
Create Kubernetes secrets
Customize Helm chart values
Deploy the Helm chart

Load Docker images and Helm charts

You must load the Docker images for Kubernetes to have the ability to create, manage, and run Data Catalog containers. To load them from the archive to the private registry, use the helper script.

Procedure

Log into the registry, and make sure you have the following required software to run the script:
- jq
- docker
- tar
Load the Docker images on each node of your cluster with the following example command:
./ldc-load-images.sh -r myregisty.azurecr.io --images /tmp/ldc-images-7.0.0.tar.gz
(Optional) Re-tag the loaded images and push them to your organization's Docker registry for improved manageability.

Create Kubernetes secrets

You create Kubernetes secrets to store and maintain small amounts of sensitive data, such as passwords, tokens, and keys.

Perform the following steps create secrets for configuration information in the core-site.xml file and your license files:

Procedure

Use the following example command to create a secret for your configurations:
kubectl create secret generic ldc-custom-core-site --from-file="core-site.xml"
NoteThe file must specifically be named core-site.xml.
Use the following example command to create secrets for your ldc-license-public-keystore.p12 and license-features.yaml license files:
kubectl create secret generic ldc-license --from-file=license-features.yaml --from-file=ldc-license-public-keystore.p12
NoteThese files must specifically be named ldc-license-public-keystore.p12 and license-features.yaml.

Customize Helm chart values

You will need to customize certain values that you provide to the Helm chart. Create a custom-values.yml file with contents similar to the following examples.

Perform the following steps to customize your custom-values.xml Helm deployment file:

Procedure

Create a local copy of the custom-values.xml file with the following example command:
touch custom-values.yaml

Customize the local version of your XML file similar to one of the following examples:

Minimum required configuration for services exposed via NodePort:

keycloak:
  service:
    type: NodePort
    nodePort: 31111
app-server:
  service:
    type: NodePort
    httpsNodePort: 31112
  keycloak:
    authServerUrl: "http://<k8s node host name>:31111/auth"
global:
  registry: myregisty.azurecr.io

Minimum required configuration for services exposed via the Ingress controller:

keycloak:
  ingress:
      enabled: true
      hosts:
      - host: keycloak-dev1.hv.com
        paths:
          - path: /
            pathType: Prefix
      tls:
        - hosts:
          - "keycloak-dev1.hv.com"
          secretName: keycloak-ingress-certs
app-server:
  ingress:
    enabled: true
    hosts:
    - host: app-server-dev1.hv.com
      paths:
        - path: /
          pathType: Prefix
    tls:
      - hosts:
        - "app-server-dev1.hv.com"
        secretName: app-server-ingress-certs
  keycloak:
    authServerUrl: "https://keycloak-dev1.hv.com/auth"
global:
  registry: myregisty.azurecr.io

Apply further customizations as needed based on the charts in Chart parameters.

Deploy the Helm chart

With the Helm chart customized for your system, you can deploy it to start running Data Catalog.

Perform the following steps to deploy the chart and start running Data Catalog:

Procedure

Use the following example command to deploy the chart:
helm install --wait ldc ldc-7.0.0.tgz -f custom-values.yml
Use the following example command to temporarily access Data Catalog for a deployment test:
kubectl port-forward svc/ldc-app-server 8082:8082
Open http://localhost:8082 in your local browser to test the deployment.

Next steps

To make Data Catalog generally available, you can expose the ldc-app-server service using your cluster's Ingress gateway or load balancer, depending on your system.

Chart parameters

You can customize your Helm chart for Data Catalog with the following parameters:

Agent

Parameter	Type	Description	Default value
`agent.appServerGraphQLUrl`	string		None (leave unspecified)
`agent.appServerWSUrl`	string	An empty value means the value will be generated using the template function	None (leave unspecified)
`agent.enabled`	boolean		`true`
`agent.isDefault`	boolean		`true`
`agent.serviceAccount.create`	string		true
`agent.serviceAccount.name`	string		None (leave unspecified)
`agent.spark.historyServer.address`	string		`"http://{{ .Release.Name }}-spark-history-server:18080"`
`agent.spark.historyServer.storageLocation`	string		`"s3a://spark-history/events/"`
`agent.spark.jarUpload.accessKey`	string		`"minioadmin"`
`agent.spark.jarUpload.endpoint`	string		`"http://{{ .Release.Name }}-minio-bundled:9000"`
`agent.spark.jarUpload.location`	string		`"s3a://ldc/cluster_jars"`
`agent.spark.jarUpload.secretKey`	string		`"minioadmin"`
`agent.spark.jarUpload.secretToken`	string		None (leave unspecified)
`agent.spark.k8sMasterEnabled`	boolean		`true`
`agent.spark.secure`	boolean		`true`
`agent.spark.serviceAccount`	boolean		`"{{ .Release.Name }}-spark"`

Application server

Parameter	Type	Description	Default value
`app-server.configurationOverridesExtraEnv[0].name`	string		`"MINIO_SECRET_KEY"`
`app-server.configurationOverridesExtraEnv[0].value`	string		`"minioadmin"`
`app-server.configurationOverridesExtraEnv[1].name`	string		`"MINIO_ACCESS_KEY"`
`app-server.configurationOverridesExtraEnv[1].value`	string		`"minioadmin"`
`app-server.configurationOverridesExtraEnv[2].name`	string		`"MINIO_ENDPOINT"`
`app-server.configurationOverridesExtraEnv[2].value`	string		`"http://{{ .Release.Name }}-minio-bundled:9000"`
`app-server.configurationOverrides[0].component`	string		`"__template_agent"`
`app-server.configurationOverrides[0].propertyKey`	string		`"ldc.metadata.hdfs.large_properties.attributes"`
`app-server.configurationOverrides[0].value[0]`	string		`"fs.s3a.access.key=${MINIO_ACCESS_KEY}"`
`app-server.configurationOverrides[0].value[1]`	string		`"fs.s3a.secret.key=${MINIO_SECRET_KEY}"`
`app-server.configurationOverrides[0].value[2]`	string		`"fs.s3a.endpoint=${MINIO_ENDPOINT}"`
`app-server.configurationOverrides[0].value[3]`	string		`"fs.s3a.path.style.access=true"`
`app-server.configurationOverrides[0].value[4]`	string		`"fs.s3a.threads.max=40"`
`app-server.configurationOverrides[0].value[5]`	string		`"fs.s3a.connection.maximum=200"`
`app-server.debug`	boolean	Print debug messages in log	`false`
`app-server.enabled`	boolean		`true`
`app-server.keycloak.authPass`	string	Password for role syncing	`"admin"`
`app-server.keycloak.authServerUrl`	string	Base URL for you Realm authorization endpoint. Needs to be accessible for client's browser	`"http://localhost:8080/auth"`
`app-server.keycloak.authUser`	string	User name for role syncing	`"admin"`
`app-server.keycloak.callbackUrl`	string	URL to which Keycloak will redirect the user after granting authentication. By default is it relative, but it could be an absolute URL	`"/callback"`
`app-server.keycloak.clientID`	string	This should match your Application Name, resource or OAuth Client Name.	`"ldc-client"`
`app-server.keycloak.realm`	string	Name of your Keycloak realm	`"ldc-realm"`
`app-server.keycloak.resource`	string	This should match your Application Name, resource or OAuth Client Name.	`"ldc-client"`
`app-server.mongodbURI`	string	`nil`

Miscellaneous

Parameter	Type	Description	Default value
`global.registry`	string	Override registry for Hitachi Vantara-managed images. By default, they are in ldmp-docker.repo.orl.eng.hitachivantara.com	`nil`
`keycloak.enabled`	boolean	Whatever to deploy or not dev/demo keycloak instance	`true`
`minio-bundled`	object	Minio helm chart config	For reference see MinIO Helm chart
`minio-bundled.enabled`	boolean	Whatever to deploy or not dev/demo minio instance	`true`
`mongodb.enabled`	boolean	Whatever to deploy or not dev/demo mongodb instance	`true`
`spark-history-server.enabled`	boolean		`true`
`tekton-hooks.enabled`	boolean		`false`

Pentaho+ documentation has moved!

The new product documentation portal is here. Check it out now at docs.hitachivantara.com.

Before you begin

Installing on Kubernetes

Load Docker images and Helm charts

Create Kubernetes secrets

Customize Helm chart values

Deploy the Helm chart

Chart parameters