Skip to main content

Pentaho+ documentation has moved!

The new product documentation portal is here. Check it out now at docs.hitachivantara.com

 

Hitachi Vantara Lumada and Pentaho Documentation

Resources

Parent article

Use the following supplemental resources as needed for your installation.

Setting up a local Docker registry

If you do not have access to an organization container registry, it is possible to make your own registry using Docker. This registry must be set up in a location that has:

  • Docker installed (the latest stable release will suffice).
  • Internet access to pull the registry image.
  • Accessible from your Kubernetes host.

Run the command to create a Docker registry on your server, which will create a registry accessible on port 5000.

docker run -d -p 5000:5000 --name registry registry:2

Using the ldc-load-images.sh script, you can now push the Hitachi Vantara Docker images to the local registry which runs on <hostname>:5000.

./ldc-load-images.sh -r <hostname>:5000 --images <path to ldc-images TAR GZ file>

Set up a Data Catalog instance for evaluation

You can set up a Data Catalog instance for evaluation or demonstration purposes by using a private registry. By using a private registry, you are eliminating the set up of a secure registry, which changes the installation process since the environment is not used for production purposes.

ImportantA private registry is for demonstration purposes and should not be used in a production environment.

To set up a private registry, you must first deploy an insecure local registry in docker. For more information see: Test an insecure registry.

Using the ldc-load-images.sh script, you can push the Docker images to the local registry that runs on <localhost>:5000. Optionally, Docker Daemon can be configured to work with remote insecure registry. For more information, see https://docs.docker.com/registry/insecure/#deploy-a-plain-http-registry.

Once the registry is created, and the Data Catalog images are loaded on it, you can use the following configuration as an example:

keycloak:
2  service:
3    type: NodePort
4    nodePort: 30880
5    httpsType: NodePort
6    httpsNodePort: 30843
7app-server:
8  service:
9    type: NodePort
10    nodePort: 31080
11    httpsNodePort: 31083
12    wsNodePort: 31083
13  keycloak:
14    authServerUrl: "https://<ip-address>:30843"
15  ingress:
16    enabled: true
17    hosts:
18    - paths:
19      - path: /
20        pathType: Prefix
21  untrustedCertsPolicy: ALLOW
22minio-bundled:
23  buckets: [{"name": "ldc", "policy": "none"}, {"name": "ldc-demo-data", "policy": "none"}, {"name": "spark-history", "policy": "none"}]
24  accessKey: minioadmin
25  secretKey: minioadmin
26  persistence:
27    enabled: true
28    size: 3Gi
29  service:
30    type: NodePort
31    nodePort: 30900
32  consoleService:
33    type: NodePort
34    nodePort: 30901
35mongodb:
36  service:
37    type: NodePort
38  persistence:
39    enabled: true
40    volumeClaim:
41      size: 10Gi
42      accessModes:
43        - ReadWriteOnce
44  
45agent:
46  initJDBC:
47    enabled: true
48  seedJDBC:
49    enabled: true
50    sources:
51       - http
52    http:
53     list:
54       - https://repo1.maven.org/maven2/org/postgresql/postgresql/42.3.1/postgresql-42.3.1.jar
55       - https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc8/21.1.0.0/ojdbc8-21.1.0.0.jar
56       - https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/10.2.0.jre8/mssql-jdbc-10.2.0.jre8.jar
57       - https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.27/mysql-connector-java-8.0.27.jar
58       - https://repo1.maven.org/maven2/com/vertica/jdbc/vertica-jdbc/11.0.2-0/vertica-jdbc-11.0.2-0.jar
59       - https://repo1.maven.org/maven2/net/snowflake/snowflake-jdbc/3.13.14/snowflake-jdbc-3.13.14.jar
60        
61rest-server:
62  service:
63    type: NodePort
64    nodePort: 31088
65    wsNodePort: 31089
66  keycloak:
67    authServerUrl: "https://<ip-address>/realms/ldc-realm"
68  oidc:
69    tls:
70      verification: none
71  okhttp:
72    trustUnknownCerts: true
73  ingress:
74    enabled: true
75    hosts:
76    - paths:
77        - path: /api/v1
78          pathType: Prefix
79        - path: /swagger-ui
80          pathType: Prefix
81        - path: /api-docs
82          pathType: Prefix
83global:
84  registry: localhost:5000

Helm chart values

Use the following tables to determine the Helm chart values for Lumada Data Catalog 7.3. If you need Helm chart values for Data Catalog 7.2, see https://help.hitachivantara.com/Documentation/Lumada/Lumada_Data_Catalog/7.2/Install/Resources#Helm_chart_values.

Agent values

Use the following table to determine Helm chart values for the Data Catalog Agent. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:

KeyTypeDescriptionDefault
affinityobject{}
appServerGraphQLUrlstringA graphql endpoint, could be template function"http://{{ .Release.Name }}-app-server:8080/graphql-api/"
appServerWSUrlstringWeb socket endpoint, could be template function"ws://{{ .Release.Name }}-app-server:8080/wsagent"
extraCertsstringExtra CA certificates in PEM formatnil
image.namestring"lumada-catalog/agent"
image.pullPolicystring"IfNotPresent"
image.repositorystring"ldmp-docker.repo.<your domain name>"
image.tagstring""
imagePullSecretslist[]
initialization.debugbooltrue
isDefaultboolfalse
logbooltrue
namestring"local-agent"
nodeSelectorobject{}
persistenceobject{"enabled":false,"logSubPath":"log","volumeClaim":{"accessModes":["ReadWriteOnce"],"annotations":{},"size":"1Gi"}}
podAnnotationsobject{}
podSecurityContext.fsGroupint1000
podSecurityContext.runAsUserint1000
podSecurityContextSeedJDBC.runAsUserint1000
replicaCountint1
resourcesobject{}
securityContextobject{}
seedJDBCobjectSeed jdbc drivers from an external source (http, s3, adls){"adls":{"accountKey":null,"accountName":"caasdata","existingSecretName":null,"fileSystem":"ldc-data","path":"/ext/jdbc"},"enabled":true,"http":{"list":null},"s3":{"accessKey":"minioadmin","bucket":"ldc-discovery-cache","endpoint":"http://{{ .Release.Name }}-minio-bundled:9000","existingSecretName":null,"path":"/ext/jdbc","secretKey":"<secretKey>"},"sources":["s3","http"]}
seedJDBC.http.liststringArray of linksnil
serviceAccount.annotationsobjectAnnotations to add to the service account{}
serviceAccount.createboolSpecifies whether a service account should be created.true
serviceAccount.namestringThe name of the service account to use. If not set and serviceAccount.create is true, a name is generated using the fullname template.nil
spark.disabeIstioSideCarboolfalse
spark.historyServer.pathstring"/events"
spark.historyServer.urlstring"http://{{ .Release.Name }}-spark-history-server:18080"
spark.jarCleanup.annotationsobject{}
spark.jarCleanup.cronSchedulestringHow often to run a clean up job"0 0 * * *"
spark.jarCleanup.deleteAfterDaysintNumber of days to keep the JAR file upload2
spark.jarCleanup.failedJobsHistoryLimitint 2
spark.jarCleanup.filesFilterstringA grep command argument"spark-upload-"
spark.jarCleanup.podSecurityContextstring {}
spark.jarCleanup.resourcesobject{}
spark.jarCleanup.successfulJobsHistoryLimitint 0
spark.jarUpload.pathstring"/cluster_jars"
aspark.k8sMasterEnabledbooltrue
spark.k8sMasterUrlstring"k8s://https://kubernetes.default.svc:443"
spark.resources.driver.memorystring"2048m"
spark.resources.executor.memorystring"1536m"
spark.securebooltrue
spark.serviceAccountstring"{{ .Release.Name }}-spark"
spark.storageobjectStorage configuration{"adls":{"accountKey":null,"accountName":"caasdata","existingSecretName":null,"fileSystem":"ldc-data"},"s3":{"accessKey":"minioadmin","bucket":"spark-history","endpoint":"http://{{ .Release.Name }}-minio-bundled:9000","existingSecretName":null,"secretKey":"<secretKey>","sessionToken":""}}
spark.storage.adls.accountKeystringAzure Data Lake Storage (ADLS) access key for SharedKey Authnil
spark.storage.adls.accountNamestring ADLS account name"caasdata"
spark.storage.adls.fileSystemstringADLS name of the container"ldc-data"
spark.storage.s3.accessKeystringMinio/AWS s3 access key"minioadmin"
spark.storage.s3.endpointstringMinio/AWS s3 endpoint"http://{{ .Release.Name }}-minio-bundled:9000"
spark.storage.s3.secretKeystring Minio/AWS s3 secret key"<secretKey>"
NoteContact your Customer Success Engineer with any questions on default credentials.
spark.storage.s3.sessionTokenstringAWS s3 session token (optional)""
spark.storageNamestringStorage name for historyServer and jarUpload. Supported values are s3 and adls."s3"
tolerationslist[]
trustStorestring"/opt/ldc/agent/conf/ldc-truststore"
trustStorePasswordstring"<password>"
NoteContact your Customer Success Engineer with any questions on default credentials.
Application Server values

Use the following table to determine Helm chart values for the Data Catalog Application Server. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:

KeyTypeDescriptionDefault
affinityobject{}
configurationOverrideslist[]
configurationOverridesExtraEnvstringnil
debugboolPrint debug messages in logfalse
environmentTypestringType of the environment on which the application is running. Value can be one of these values: STANDALONE, LDOS, SAAS"STANDALONE"
extraCertsstringExtra certificates in PEM format passed to the nodejs processnil
extraEnvstringExtra environment in k8s form passed to the nodejs processnil
graphql.autoscaling.enabledboolEnable Horizontal Pod Autoscaler (HPA) for the graphql processfalse
graphql.autoscaling.maxReplicasintMaximum number of replicas for the graphql (HPA) process5
graphql.autoscaling.minReplicasintMinimum number of replicas for the graphql (HPA) process1
graphql.autoscaling.targetCPUUtilizationPercentageintAverage CPU utilization percentage for the graphql (HPA) process70
graphql.autoscaling.targetMemoryUtilizationPercentagestringAverage memory utilization percentage for the graphql (HPA) processnil
graphql.replicaCountint1
idpStrategyConfigobjectIDP strategy configuration map name{}
image.namestring"lumada-catalog/app-server"
image.pullPolicystring"IfNotPresent"
image.repositorystring"ldmp-docker.repo.<your domain name>"
image.tagstring""
imageKubectl.namestring"bitnami/kubectl"
imageKubectl.pullPolicystring"IfNotPresent"
imageKubectl.repositorystring"docker.io"
imageKubectl.tagstring"1.22.15"
imageMongodbMigration.namestring"lumada-catalog/mongodb-migration-tool"
imageMongodbMigration.pullPolicystring"IfNotPresent"
imageMongodbMigration.repositorystring"ldmp-docker.repo.<your domain name>"
imageMongodbMigration.tagstring""
imageProxy.namestring"nginxinc/nginx-unprivileged"
imageProxy.pullPolicystring"IfNotPresent"
imageProxy.repositorystring"docker.io"
imageProxy.tagstring"1.22.1"
imagePullSecretslist[]
ingress.annotationsobject{}
ingress.classNamestring""
ingress.enabledboolfalse
ingress.hosts[0].hoststring"chart-example.local"
ingress.hosts[0].paths[0].pathstring"/"
ingress.hosts[0].paths[0].pathTypestring"ImplementationSpecific"
ingress.tlslist[]
jwtPrivateKeystringnil
jwtSecretstringnil
keycloak.authPassstringPassword for role syncing"<password>"
NoteContact your Customer Success Engineer with any questions on default credentials.
keycloak.authServerUrlstringBase URL for your Realm authorization endpoint. Needs to be accessible by the user's browser."https://localhost:8443"
keycloak.authUserstringUser name for role syncing"apiuser"
keycloak.callbackUrlstringURL to which Keycloak will redirect the user after granting authentication. By default is it relative, but it could be an absolute URL."/callback"
keycloak.clientIDstringThis will match your Application Name, resource, or OAuth Client Name."ldc-client"
keycloak.clientSecretstringThis will match your OAuth client secret.nil
keycloak.existingAuthUserSecretNamestringnil
keycloak.existingClientSecretNamestringnil
keycloak.realmstringName of your Keycloak realm"ldc-realm"
keystorePasswordstringnil
keystoreSecretstringnil
metrics.enabledboolExpose Prometheus-compatible metrics on /metrics endpointfalse
metrics.serviceMonitor.enabledbool Create ServiceMonitor CR for Prometheus operatorfalse
metrics.serviceMonitor.namespacestringNamespace of Prometheus operator"prom-stack"
mongodbURIstringnil
mongodbURISecretstring"{{ .Release.Name }}-mng-cred"
mongoose.socketTimeoutMSint How long after initial connection the MongoDB driver will wait before killing a socket due to inactivity. See https://mongoosejs.com/docs/5.x/docs/connections.html for more information.30000
nodeOptionsstringnodejs optionsfor reference, see nodejs options
nodeSelectorobject{}
podAnnotationsobject{}
podSecurityContextobject{}
replicaCountint1
resourcesobject{}
securityContext.runAsUserint1000
seedKeystoreJob.podAnnotationsobject{}
service.httpsNodePortint31083#
service.httpsPortint8443
service.nodePortint31080#
service.portint8080
service.typestring"ClusterIP"
serviceAccount.annotationsobject{}
serviceAccount.createbooltrue
serviceAccount.namestring""
sessionTimeoutintUser session timeout1800
tolerationslist[]
untrustedCertsPolicystring"REJECT"

# See Port and firewall requirements in System requirements for updated port details.

Keycloak values

Use the following table to determine Helm chart values for Keycloak. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:

KeyTypeDescriptionDefault
affinityobject{}
image.namestring"keycloak/keycloak"
image.pullPolicystring"IfNotPresent"
image.repositorystring"quay.io"
image.tagstring""
imagePullSecretslist[]
ingress.annotationsobject{}
ingress.classNamestring""
ingress.enabledboolfalse
ingress.hosts[0].hoststring"chart-example.local"
ingress.hosts[0].paths[0].pathstring"/"
ingress.hosts[0].paths[0].pathTypestring"ImplementationSpecific"
ingress.tlslist[]
initialAdminUserNamestring"admin"
initialAdminUserPasswordstring"<password>"
NoteContact your Customer Success Engineer with any questions on default credentials.
nodeSelectorobject{}
persistenceobjectPersist h2 database to a persistent volume. Enable persistence using Persistent Volume Claims. See http://kubernetes.io/docs/user-guide/persistent-volumes/ for more information.{"VolumeName":"","accessModes":["ReadWriteOnce"],"annotations":{},"enabled":false,"existingClaim":"","size":"5Gi","storageClass":"","subPath":""}
persistence.existingClaimstringA manually-managed Persistent Volume and Claim (PVC) requires persistence.enabled: true. If defined, PVC must be created manually before volume will be bound.""
persistence.storageClassstringkeycloak data Persistent Volume Storage Class If defined, storageClassName: If set to "-", storageClassName: "", which disables dynamic provisioning. If undefined (the default) or set to null, no storageClassName spec is set, choosing the default provisioner. (gp2 on AWS, standard on GKE, AWS & OpenStack) Storage class of PV to bind. By default, it looks for standard storage class. If the PV uses a different storage class, specify that here.""
persistence.subPathstringIf subPath is set, mount a subfolder of a volume instead of the root of the volume. This is especially handy for volume plugins that don't natively support sub-mounting (like glusterfs).""
podAnnotationsobject{}
podSecurityContextobject{}
replicaCountint1
resourcesobjectAs a best practice, do not specify default resources, but leave this as a conscious choice for the user. This also increases the chance charts will run on environments with few resources, such as Minikube. If you want to specify resources, uncomment the lines after "resources", adjust them as necessary, and remove the curly braces after "resources".{}
securityContextobject{}
service.httpsNodePortint30843#
service.httpsPortint8443
service.httpsTypestring"ClusterIP"
service.nodePortint30880#
service.portint8080
service.typestring"ClusterIP"
serviceAccount.annotationsobject{}
serviceAccount.createbooltrue
serviceAccount.namestring""
tolerationslist[]

# See Port and firewall requirements in System requirements for updated port details.

MongoDB

Use the following table to determine Helm chart values for MongoDB. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:

KeyTypeDefault
affinityobject{}
credentailsSecretstringnil
databaseNamestring"ldcdb"
fullnameOverridestring""
image.namestring"lumada-catalog/mongodb-ee"
image.pullPolicystring"IfNotPresent"
image.repositorystring"ldmp-docker.repo.<your domain name>"
image.tagstring""
imageKubectl.namestring"bitnami/kubectl"
imageKubectl.pullPolicystring"IfNotPresent"
imageKubectl.repositorystring"docker.io"
imageKubectl.tagstring"1.22.15"
imagePullSecretslist[]
imageSideCar.namestring"cvallance/mongo-k8s-sidecar"
imageSideCar.pullPolicystring"IfNotPresent"
imageSideCar.repositorystring"docker.io"
imageSideCar.tagstring"latest"
nameOverridestring""
nodeSelectorobject{}
passwordstringnil
persistence.enabledboolfalse
persistence.volumeClaim.accessModes[0]string"ReadWriteOnce"
persistence.volumeClaim.existingClaimstringnil
persistence.volumeClaim.sizestring"10Gi"
persistence.volumeClaim.storageClassstring""
podAnnotationsobject{}
podSecurityContext.fsGroupint999
podSecurityContext.runAsUserint999
replicaCountint1
resourcesobject{}
securityContextobject{}
securityContextSideCar.runAsUserint1000
service.nodePortint30017
service.portint27017
service.typestring"ClusterIP"
serviceAccount.annotationsobject{}
serviceAccount.createbooltrue
serviceAccount.namestring""
tolerationslist[]
usernamestring"root"
REST Server values

Use the following table to determine Helm chart values for the Data Catalog REST Server. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:

KeyTypeDescriptionDefault
appserverUrlstring"http://{{ .Release.Name }}-app-server:8080"
defaultLocalestringIf the value is not set, it uses en-US by default.""
image.namestring"lumada-catalog/rest-server-native"
image.pullPolicystring"IfNotPresent"
image.repositorystring"ldmp-docker.repo.<your domain name>"
image.tagstring""
imagePullSecretslist[]
ingress.annotationsobject{}
ingress.classNamestring""
ingress.enabledboolfalse
ingress.hosts[0].hoststring"chart-example.local"
ingress.hosts[0].paths[0].pathstring"/"
ingress.hosts[0].paths[0].pathTypestring"ImplementationSpecific"
ingress.tlslist[]
initialization.debugboolfalse
keycloak.authServerUrlstringBase URL for your Realm authorization endpoint. Needs to be accessible by the user's browser."https://localhost:8443/auth/realms/ldc-realm"
keycloak.clientIDstringThis will match your Application Name, resource, or OAuth Client Name."ldc-client"
keycloak.clientSecretstringThis will match your OAuth Client Secret.nil
keycloak.existingClientSecretNamestringnil
keystoreSecretstringnil
mongodbURIstringnil
mongodbURISecretstring"{{ .Release.Name }}-mng-cred"
msgPropertiesConfigMapNamestringValidation message files configuration map name""
oidc.tls.verificationstringCertificate validation and hostname verification for OIDC provider, which can be one of these values: required, certificate-validation, or none."required"
okhttp.trustUnknownCertsboolCertificate validation for http clientfalse
podAnnotationsobject{}
podSecurityContext.fsGroupint1000
podSecurityContext.runAsUserint1000
replicaCountint1
securityContextobject{}
service.httpsNodePortint31889
service.httpsPortint8443
service.nodePortint31888
service.portint8080
service.typestring"ClusterIP"
serviceAccount.annotationsobjectAnnotations to add to the service account.{}
serviceAccount.createboolSpecifies whether a service account should be created.true
serviceAccount.namestringThe name of the service account to use. If not set and serviceAccount.create is true, a name is generated using the fullname template.nil
Spark history server values

Use the following table to determine Helm chart values for the Spark history server. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:

KeyTypeDescriptionDefault
failedCronJobHistory.cronSchedulestring"1 0 * * *"
failedCronJobHistory.failedJobsHistoryLimitint2
failedCronJobHistory.successfulJobsHistoryLimitint0
historyServer.coreSite.enableCoreSiteboolfalse
historyServer.coreSite.namestring""
historyServer.eventLogPathstring"/events"
historyServer.events.adls.accountKeystringnil
historyServer.events.adls.accountNamestring"caasdata"
historyServer.events.adls.existingSecretNamestringnil
historyServer.events.adls.fileSystemstring"ldc-data"
historyServer.events.s3.accessKeystring"minioadmin"
historyServer.events.s3.bucketstring"spark-history"
historyServer.events.s3.createBucketbooltrue
historyServer.events.s3.endpointstring"http://{{ .Release.Name }}-minio-bundled:9000"
historyServer.events.s3.existingSecretNamestringnil
historyServer.events.s3.secretKeystring"<secretKey>"
NoteContact your Customer Success Engineer with any questions on default credentials.
historyServer.events.s3.sessionTokenstring""
historyServer.eventsCreateFoldersbooltrue
historyServer.eventsStorageNamestring"s3"
historyServer.historyPathstring"/events"
historyServer.ingress.annotationsobject{}
historyServer.ingress.classNamestring""
historyServer.ingress.enabledboolfalse
historyServer.ingress.hosts[0].hoststring"chart-example.local"
historyServer.ingress.hosts[0].paths[0].pathstring"/"
historyServer.ingress.hosts[0].paths[0].pathTypestring"ImplementationSpecific"
historyServer.ingress.tlslist[]
historyServer.requestHeaderSizestring"20k"
image.namestring"lumada-catalog/spark"
image.pullPolicystring"IfNotPresent"
image.repositorystring"ldmp-docker.repo.<your domain name>"
image.tagstring"3.1.3-1.0.15"
initContainer.backoffLimitint3
initContainer.mcImage.namestring"minio/mc"
initContainer.mcImage.pullPolicystring"IfNotPresent"
initContainer.mcImage.repositorystring"docker.io"
initContainer.mcImage.tagstring"RELEASE.2021-02-14T04-28-06Z"
initContainer.podAnnotationsobject{}
initContainer.podSecurityContextobject{}
podAnnotationsobject{}
replicaCountint1
resourcesobject{}
securityContext.runAsUserint1500
securityContextCreateBucket.runAsUserint65534
service.portint18080
service.typestring"ClusterIP"
serviceAccount.annotationsobject{}
serviceAccount.createbooltrue
serviceAccount.namestring"spark"
sparkJobCleanupHook.namestring"bitnami/kubectl"
sparkJobCleanupHook.pullPolicystring"IfNotPresent"
sparkJobCleanupHook.repositorystring"docker.io"
sparkJobCleanupHook.tagstring"1.22.15"
succeededCronJobHistory.cronSchedulestring"*/4 * * * *"
succeededCronJobHistory.failedJobsHistoryLimitint2
succeededCronJobHistory.successfulJobsHistoryLimitint0
Miscellaneous values

Use the following table to determine miscellaneous Helm chart values. You can customize your Helm chart by overriding any of the following parameters in your custom-values.yml file:

KeyTypeDescriptionDefault
global.isSaasEnvboolSet to true for SaaS environmentfalse
global.registrystringOverride registry for Hitachi Vantara-managed images. By default, they are in ldmp-docker.repo.<your domain name>.nil
keycloak.enabledboolEnabled only for development or demonstration purposes.
NoteThis value should not be used for Production Data Catalog installations.
true
minio-bundledobjectMinio helm chart configurationSee vanilla minio chart for reference.
minio-bundled.enabledboolEnabled only for development or demonstration purposes.
NoteThis value should not be used for Production Data Catalog installations.
true
mongodb.enabledboolEnabled only for development or demonstration purposes.
NoteThis value should not be used for Production Data Catalog installations.
true
spark-history-server.historyServer.events.s3.createBucketboolCreate bucket on startup, working only with minio.true
tekton-hooks.enabledboolfalse