Resources
Use the following supplemental resources as needed for your installation.
Setting up a local Docker registry
If you do not have access to an organization container registry, it is possible to make your own registry using Docker. This registry must be set up in a location that has:
- Docker installed (the latest stable release will suffice).
- Internet access to pull the registry image.
- Accessible from your Kubernetes host.
Run the command to create a Docker registry on your server, which will create a registry accessible on port 5000.
docker run -d -p 5000:5000 --name registry registry:2
Using the ldc-load-images.sh script, you can now push the Hitachi Vantara Docker images to the local registry which runs on <hostname>:5000.
./ldc-load-images.sh -r <hostname>:5000 --images <path to ldc-images TAR GZ file>
Set up a Data Catalog instance for evaluation
You can set up a Data Catalog instance for evaluation or demonstration purposes by using a private registry. By using a private registry, you are eliminating the set up of a secure registry, which changes the installation process since the environment is not used for production purposes.
To set up a private registry, you must first deploy an insecure local registry in docker. For more information see: Test an insecure registry.
Using the ldc-load-images.sh
script, you can push the Docker images to the local registry that runs on <localhost>:5000
. Optionally, Docker Daemon can be configured to work with remote insecure registry. For more information, see https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry.
Once the registry is created, and the Data Catalog images are loaded on it, you can use the following configuration as an example:
keycloak: 2 service: 3 type: NodePort 4 nodePort: 30880 5 httpsType: NodePort 6 httpsNodePort: 30843 7app-server: 8 service: 9 type: NodePort 10 nodePort: 31080 11 httpsNodePort: 31083 12 wsNodePort: 31083 13 keycloak: 14 authServerUrl: "https://<ip-address>:30843" 15 ingress: 16 enabled: true 17 hosts: 18 - paths: 19 - path: / 20 pathType: Prefix 21 untrustedCertsPolicy: ALLOW 22minio-bundled: 23 buckets: [{"name": "ldc", "policy": "none"}, {"name": "ldc-demo-data", "policy": "none"}, {"name": "spark-history", "policy": "none"}] 24 accessKey: minioadmin 25 secretKey: minioadmin 26 persistence: 27 enabled: true 28 size: 3Gi 29 service: 30 type: NodePort 31 nodePort: 30900 32 consoleService: 33 type: NodePort 34 nodePort: 30901 35mongodb: 36 service: 37 type: NodePort 38 persistence: 39 enabled: true 40 volumeClaim: 41 size: 10Gi 42 accessModes: 43 - ReadWriteOnce 44 45agent: 46 initJDBC: 47 enabled: true 48 seedJDBC: 49 enabled: true 50 sources: 51 - http 52 http: 53 list: 54 - https://repo1.maven.org/maven2/org/postgresql/postgresql/42.3.1/postgresql-42.3.1.jar 55 - https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc8/21.1.0.0/ojdbc8-21.1.0.0.jar 56 - https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/10.2.0.jre8/mssql-jdbc-10.2.0.jre8.jar 57 - https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.27/mysql-connector-java-8.0.27.jar 58 - https://repo1.maven.org/maven2/com/vertica/jdbc/vertica-jdbc/11.0.2-0/vertica-jdbc-11.0.2-0.jar 59 - https://repo1.maven.org/maven2/net/snowflake/snowflake-jdbc/3.13.14/snowflake-jdbc-3.13.14.jar 60 61rest-server: 62 service: 63 type: NodePort 64 nodePort: 31088 65 wsNodePort: 31089 66 keycloak: 67 authServerUrl: "https://<ip-address>/realms/ldc-realm" 68 oidc: 69 tls: 70 verification: none 71 okhttp: 72 trustUnknownCerts: true 73 ingress: 74 enabled: true 75 hosts: 76 - paths: 77 - path: /api/v1 78 pathType: Prefix 79 - path: /swagger-ui 80 pathType: Prefix 81 - path: /api-docs 82 pathType: Prefix 83global: 84 registry: localhost:5000
Helm chart values
Use the following tables to determine the Helm chart values for Lumada Data Catalog 7.2. If you need Helm chart values for Data Catalog 7.3, see https://help.hitachivantara.com/Documentation/Lumada/Lumada_Data_Catalog/7.3/Install/Resources#Helm_chart_values.
Use the following table to determine Helm chart values for the Data Catalog Agent. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:
Key | Type | Description | Default |
affinity | object | {} | |
appServerGraphQLUrl | string | nil | |
appServerWSUrl | string | Empty value means value is generated using template function | nil |
image.name | string | "lumada-catalog/agent" | |
image.pullPolicy | string | "IfNotPresent" | |
image.repository | string | "ldmp-docker.repo.<your domain name>" | |
image.tag | string | "" | |
imagePullSecrets | list | [] | |
initialization.debug | bool | true | |
isDefault | bool | true | |
log | bool | true | |
name | string | "local-agent" | |
nodeSelector | object | {} | |
persistence.enabled | bool | false | |
persistence.logSubPath | string | '"log" | |
persistence.volumeClaim.accessModes[0] | string | "ReadWriteOnce" | |
persistence.volumeClaim.annotations | object | {} | |
persistence.volumeClaim.size | string | "1Gi" | |
podAnnotations | object | {} | |
podSecurityContext.fsGroup | int | 1000 | |
podSecurityContext.runAsUser | int | 1000 | |
podSecurityContextSeedJDBC.runAsUser | int | 1000 | |
replicaCount | int | 1 | |
resources | object | {} | |
securityContext | object | {} | |
seedJDBC.adls.accountKey | string | nil | |
seedJDBC.adls.accountName | string | "caasdata" | |
seedJDBC.adls.existingSecretName | string | nil | |
seedJDBC.adls.fileSystem | string | "ldc-data" | |
seedJDBC.adls.path | string | "/ext/jdbc" | |
seedJDBC.enabled | bool | true | |
seedJDBC.http.list | string | nil | |
seedJDBC.s3.accessKey | string | "minioadmin" | |
seedJDBC.s3.bucket | string | "ldc-discovery-cache" | |
seedJDBC.s3.endpoint | string | "http://{{ .Release.Name }}-minio-bundled:9000" | |
seedJDBC.s3.existingSecretName | string | Nil | |
seedJDBC.s3.path | string | "/ext/jdbc" | |
seedJDBC.s3.secretKey | string | "<secretKey>" NoteContact your Customer Success Engineer with any questions on default credentials. | |
seedJDBC.sources[0] | string | "s3" | |
seedJDBC.sources[1] | string | "http" | |
serviceAccount.annotations | object | {} | |
serviceAccount.create | bool | True | |
serviceAccount.name | string | Nil | |
spark.disabeIstioSideCar | bool | False | |
spark.historyServer.path | string | "/events" | |
spark.historyServer.url | string | "http://{{ .Release.Name }}-spark-history-server:18080" | |
spark.jarCleanup.annotations | object | {} | |
spark.jarCleanup.cronSchedule | string | "0 0 * * *" | |
spark.jarCleanup.deleteAfterDays | int | 2 | |
spark.jarCleanup.failedJobsHistoryLimit | int | 2 | |
spark.jarCleanup.filesFilter | string | "spark-upload-" | |
spark.jarCleanup.podSecurityContext | string | {} | |
spark.jarCleanup.resources | object | {} | |
spark.jarCleanup.successfulJobsHistoryLimit | int | 0 | |
spark.jarUpload.path | string | "/cluster_jars" | |
aspark.k8sMasterEnabled | bool | True | |
spark.k8sMasterUrl | string | "k8s://https://kubernetes.default.svc:443" | |
spark.resources.driver.memory | string | "2048m" | |
spark.resources.executor.memory | string | "1536m" | |
spark.secure | bool | true | |
spark.serviceAccount | string | "{{ .Release.Name }}-spark" | |
spark.storage.adls.accountKey | string | Nil | |
spark.storage.adls.accountName | string | "caasdata" | |
spark.storage.adls.existingSecretName | string | Nil | |
spark.storage.adls.fileSystem | string | "ldc-data" | |
spark.storage.s3.accessKey | string | "minioadmin" | |
spark.storage.s3.bucket | string | "spark-history" | |
spark.storage.s3.endpoint | string | "http://{{ .Release.Name }}-minio-bundled:9000" | |
spark.storage.s3.existingSecretName | string | nil | |
spark.storage.s3.secretKey | string | "<secretKey>" NoteContact your Customer Success Engineer with any questions on default credentials. | |
spark.storage.s3.sessionToken | string | "" | |
spark.storageName | string | "s3" | |
tolerations | list | [] |
Use the following table to determine Helm chart values for the Data Catalog Application Server. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:
Key | Type | Description | Default |
affinity | object | {} | |
configurationOverrides | list | [] | |
configurationOverridesExtraEnv | string | nil | |
debug | bool | Print debug messages in log | false |
extraEnv | string | Extra env in k8s form passed to nodejs process | nil |
graphql.autoscaling.enabled | bool | Enable Horizontal Pod Autoscaler ([HPA]) for the graphql process | false |
graphql.autoscaling.maxReplicas | int | Maximum number of replicas for the graphql [HPA] process | 5 |
graphql.autoscaling.minReplicas | int | Minimum number of replicas for the graphql [HPA] process | 1 |
graphql.autoscaling.targetCPUUtilizationPercentage | int | Average CPU utilization percentage for the graphql [HPA] process | 70 |
graphql.autoscaling.targetMemoryUtilizationPercentage | string | Average memory utilization percentage for the graphql [HPA] process | nil |
idpStrategyConfig | object | IDP strategy configuration map name | {} |
image.name | string | "lumada-catalog/app-server" | |
image.pullPolicy | string | "IfNotPresent" | |
image.repository | string | "ldmp-docker.repo.<your domain name>" | |
image.tag | string | "" | |
imageKubectl.name | string | "bitnami/kubectl" | |
imageKubectl.pullPolicy | string | "IfNotPresent" | |
imageKubectl.repository | string | "docker.io" | |
imageKubectl.tag | string | "1.22.13" | |
imageMongodbMigration.name | string | "lumada-catalog/mongodb-migration-tool" | |
imageMongodbMigration.pullPolicy | string | "IfNotPresent" | |
imageMongodbMigration.repository | string | "ldmp-docker.repo.<your domain name>" | |
imageMongodbMigration.tag | string | "" | |
imageProxy.name | string | "nginxinc/nginx-unprivileged" | |
imageProxy.pullPolicy | string | "IfNotPresent" | |
imageProxy.repository | string | "docker.io" | |
imageProxy.tag | string | "1.22.0" | |
imagePullSecrets | list | [] | |
ingress.annotations | object | {} | |
ingress.className | string | "" | |
ingress.enabled | bool | false | |
ingress.hosts[0].host | string | "chart-example.local" | |
ingress.hosts[0].paths[0].path | string | "/" | |
ingress.hosts[0].paths[0].pathType | string | "ImplementationSpecific" | |
ingress.tls | list | [] | |
jwtPrivateKey | string | nil | |
jwtSecret | string | nil | |
keycloak.authPass | string | Password for role syncing | "<password>" NoteContact your Customer Success Engineer with any questions on default credentials. |
keycloak.authServerUrl | string | Base URL for your Realm authorization endpoint. Needs to be accessible by the user's browser. | "https://localhost:8443" |
keycloak.authUser | string | User name for role syncing | "apiuser" |
keycloak.callbackUrl | string | URL to which Keycloak will redirect the user after granting authentication. By default is it relative but could be an absolute URL. | "/callback" |
keycloak.clientID | string | This will match your Application Name, resource, or OAuth Client Name | "ldc-client" |
keycloak.clientSecret | string | This will match your OAuth Client Secret | nil |
keycloak.existingAuthUserSecretName | string | nil | |
keycloak.existingClientSecretName | string | nil | |
keycloak.realm | string | Name of your Keycloak realm | "ldc-realm" |
keystorePassword | string | nil | |
keystoreSecret | string | nil | |
metrics.enabled | bool | Expose Prometheus-compatible metrics on /metrics endpoint | false |
metrics.serviceMonitor.enabled | bool | Create ServiceMonitor CR for Prometheus operator | false |
metrics.serviceMonitor.namespace | string | Namespace of Prometheus operator | "prom-stack" |
mongodbURI | string | nil | |
mongodbURISecret | string | "{{ .Release.Name }}-mng-cred" | |
mongoose.socketTimeoutMS | int | How long after initial connection the MongoDB driver will wait before killing a socket due to inactivity. See https://mongoosejs.com/docs/5.x/docs/connections.html for more information. | 30000 |
nodeOptions | string | nodejs options | for reference see nodejs options |
nodeSelector | object | {} | |
podAnnotations | object | {} | |
podSecurityContext | object | {} | |
replicaCount | int | 1 | |
resources | object | {} | |
securityContext.runAsUser | int | 1000 | |
seedKeystoreJob.podAnnotations | object | {} | |
service.httpsNodePort | int | 31083 | |
service.httpsPort | int | 8443 | |
service.nodePort | int | 31080 | |
service.port | int | 8080 | |
service.type | string | "ClusterIP" | |
serviceAccount.annotations | object | {} | |
serviceAccount.create | bool | true | |
serviceAccount.name | string | "" | |
sessionTimeout | int | User session timeout | 1800 |
tolerations | list | [] | |
untrustedCertsPolicy | string | "REJECT" |
Use the following table to determine Helm chart values for Keycloak. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:
Key | Type | Description | Default |
affinity | object | {} | |
image.name | string | "keycloak/keycloak" | |
image.pullPolicy | string | "IfNotPresent" | |
image.repository | string | "quay.io" | |
image.tag | string | "" | |
imagePullSecrets | list | [] | |
ingress.annotations | object | {} | |
ingress.className | string | "" | |
ingress.enabled | bool | false | |
ingress.hosts[0].host | string | "chart-example.local" | |
ingress.hosts[0].paths[0].path | string | "/" | |
ingress.hosts[0].paths[0].pathType | string | "ImplementationSpecific" | |
ingress.tls | list | [] | |
nodeSelector | object | {} | |
podAnnotations | object | {} | |
podSecurityContext | object | {} | |
replicaCount | int | 1 | |
resources | object | {} | |
securityContext | object | {} | |
service.httpsNodePort | int | 30843 | |
service.httpsPort | int | 8443 | |
service.httpsType | string | "ClusterIP" | |
service.nodePort | int | 30880 | |
service.port | int | 8080 | |
service.type | string | "ClusterIP" | |
serviceAccount.annotations | object | {} | |
serviceAccount.create | bool | true | |
serviceAccount.name | string | "" | |
tolerations | list | [] |
Use the following table to determine Helm chart values for MongoDB. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:
Key | Type | Default |
affinity | object | {} |
credentailsSecret | string | nil |
databaseName | string | "ldcdb" |
fullnameOverride | string | "" |
image.name | string | "lumada-catalog/mongodb" |
image.pullPolicy | string | "IfNotPresent" |
image.repository | string | "ldmp-docker.repo.<your domain name>" |
image.tag | string | "" |
imageKubectl.name | string | "bitnami/kubectl" |
imageKubectl.pullPolicy | string | "IfNotPresent" |
imageKubectl.repository | string | "docker.io" |
imageKubectl.tag | string | "1.22.13" |
imagePullSecrets | list | [] |
imageSideCar.name | string | "cvallance/mongo-k8s-sidecar" |
imageSideCar.pullPolicy | string | "IfNotPresent" |
imageSideCar.repository | string | "docker.io" |
imageSideCar.tag | string | "latest" |
nameOverride | string | "" |
nodeSelector | object | {} |
password | string | nil |
persistence.enabled | bool | false |
persistence.volumeClaim.accessModes[0] | string | "ReadWriteOnce" |
persistence.volumeClaim.existingClaim | string | nil |
persistence.volumeClaim.size | string | "10Gi" |
persistence.volumeClaim.storageClass | string | "" |
podAnnotations | object | {} |
podSecurityContext.fsGroup | ing | 999 |
podSecurityContext.runAsUser | int | 999 |
replicaCount | int | 1 |
resources | object | {} |
securityContext | object | {} |
securityContextSideCar.runAsUser | int | 1000 |
service.nodePort | int | 30017 |
service.port | int | 27017 |
service.type | string | "ClusterIP" |
serviceAccount.annotations | object | {} |
serviceAccount.create | bool | true |
serviceAccount.name | string | "" |
tolerations | list | [] |
username | string | "root" |
Use the following table to determine Helm chart values for the Data Catalog REST Server. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:
Key | Type | Description | Default |
appserverUrl | string | "http://{{ .Release.Name }}-app-server:8080" | |
image.name | string | "lumada-catalog/rest-server-native" | |
image.pullPolicy | string | "IfNotPresent" | |
image.repository | string | "ldmp-docker.repo.<your domain name>" | |
image.tag | string | "" | |
imagePullSecrets | list | [] | |
ingress.annotations | object | {} | |
ingress.className | string | "" | |
ingress.enabled | bool | false | |
ingress.hosts[0].host | string | "chart-example.local" | |
ingress.hosts[0].paths[0].path | string | "/" | |
ingress.hosts[0].paths[0].pathType | string | "ImplementationSpecific" | |
ingress.tls | list | [] | |
initialization.debug | bool | false | |
keycloak.authServerUrl | string | Base URL for your Realm authorization endpoint. Needs to be accessible by the user's browser. | "https://localhost:8443/auth/realms/ldc-realm" |
keycloak.clientID | string | This will match your Application Name, resource, or OAuth Client Name. | "ldc-client" |
keycloak.clientSecret | string | This will match your OAuth Client Secret. | nil |
keycloak.existingClientSecretName | string | nil | |
keystoreSecret | string | nil | |
mongodbURI | string | nil | |
mongodbURISecret | string | "{{ .Release.Name }}-mng-cred" | |
oidc.tls.verification | string | Certificate validation and hostname verification for OIDC provider, which can be one of these values: required, certificate-validation, or none. | "required" |
okhttp.trustUnknownCerts | bool | Certificate validation for http client | false |
podAnnotations | object | {} | |
podSecurityContext.fsGroup | int | 1000 | |
podSecurityContext.runAsUser | int | 1000 | |
replicaCount | int | 1 | |
securityContext | object | {} | |
service.httpsNodePort | int | 31889 | |
service.httpsPort | int | 8443 | |
service.nodePort | int | 31888 | |
service.port | int | 8080 | |
service.type | string | "ClusterIP" | |
serviceAccount.annotations | object | Annotations to add to the service account. | {} |
serviceAccount.create | bool | Specifies whether a service account should be created. | true |
serviceAccount.name | string | The name of the service account to use. If not set and serviceAccount.create is true, a name is generated using the fullname template. | nil |
Use the following table to determine Helm chart values for the Spark history server. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:
Key | Type | Description | Default |
failedCronJobHistory.cronSchedule | string | "1 0 * * *" | |
failedCronJobHistory.failedJobsHistoryLimit | int | 2 | |
failedCronJobHistory.successfulJobsHistoryLimit | int | 0 | |
historyServer.coreSite.enableCoreSite | bool | false | |
historyServer.coreSite.name | string | "" | |
historyServer.eventLogPath | string | "/events" | |
historyServer.events.adls.accountKey | string | nil | |
historyServer.events.adls.accountName | string | "caasdata" | |
historyServer.events.adls.existingSecretName | string | nil | |
historyServer.events.adls.fileSystem | string | "ldc-data" | |
historyServer.events.s3.accessKey | string | "minioadmin" | |
historyServer.events.s3.bucket | string | "spark-history" | |
historyServer.events.s3.createBucket | bool | true | |
historyServer.events.s3.endpoint | string | "http://{{ .Release.Name }}-minio-bundled:9000" | |
historyServer.events.s3.existingSecretName | string | nil | |
historyServer.events.s3.secretKey | string | "<secretKey>" NoteContact your Customer Success Engineer with any questions on default credentials. | |
historyServer.events.s3.sessionToken | string | "" | |
historyServer.eventsCreateFolders | bool | true | |
historyServer.eventsStorageName | string | "s3" | |
historyServer.historyPath | string | "/events" | |
historyServer.ingress.annotations | object | {} | |
historyServer.ingress.className | string | "" | |
historyServer.ingress.enabled | bool | false | |
historyServer.ingress.hosts[0].host | string | "chart-example.local" | |
historyServer.ingress.hosts[0].paths[0].path | string | "/" | |
historyServer.ingress.hosts[0].paths[0].pathType | string | "ImplementationSpecific" | |
historyServer.ingress.tls | list | [] | |
historyServer.requestHeaderSize | string | "20k" | |
image.name | string | "lumada-catalog/spark" | |
image.pullPolicy | string | "IfNotPresent" | |
image.repository | string | "ldmp-docker.repo.<your domain name>" | |
image.tag | string | "3.1.3-1.0.12" | |
initContainer.backoffLimit | int | 3 | |
initContainer.mcImage.name | string | "minio/mc" | |
initContainer.mcImage.pullPolicy | string | "IfNotPresent" | |
initContainer.mcImage.repository | string | "docker.io" | |
initContainer.mcImage.tag | string | "RELEASE.2021-02-14T04-28-06Z" | |
initContainer.podAnnotations | object | {} | |
initContainer.podSecurityContext | object | {} | |
podAnnotations | object | {} | |
replicaCount | int | 1 | |
resources | object | {} | |
securityContext.runAsUser | int | 1500 | |
securityContextCreateBucket.runAsUser | int | 65534 | |
service.port | int | 18080 | |
service.type | string | "ClusterIP" | |
serviceAccount.annotations | object | {} | |
serviceAccount.create | bool | true | |
serviceAccount.name | string | "spark" | |
sparkJobCleanupHook.name | string | "bitnami/kubectl" | |
sparkJobCleanupHook.pullPolicy | string | "IfNotPresent" | |
sparkJobCleanupHook.repository | string | "docker.io" | |
sparkJobCleanupHook.tag | string | "1.22.13" | |
succeededCronJobHistory.cronSchedule | string | "*/4 * * * *" | |
succeededCronJobHistory.failedJobsHistoryLimit | int | 2 | |
succeededCronJobHistory.successfulJobsHistoryLimit | int | 0 |
Use the following table to determine miscellaneous Helm chart values. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:
Key | Type | Description | Default |
global.isSaasEnv | bool | Set to true for SaaS environment | false |
global.registry | string | Override registry for Hitachi Vantara-managed images. By default, they are in ldmp-docker.repo.<your domain name> | nil |
keycloak.enabled | bool | Enabled only for development or demonstration purposes. NoteThis value should not be used for Production Data Catalog installations. | true |
minio-bundled | object | minio | for reference use vanilla minio chart |
minio-bundled.enabled | bool | Enabled only for development or demonstration purposes. NoteThis value should not be used for Production Data Catalog installations. | true |
mongodb.enabled | bool | Enabled only for development or demonstration purposes. NoteThis value should not be used for Production Data Catalog installations. | true |
spark-history-server.historyServer.events.s3.createBucket | bool | Create bucket on startup, working only with minio . | true |
tekton-hooks.enabled | bool | false |