Installation on Kubernetes

Large Properties configuration determines where metadata from your job runs are stored. By default, the large properties location is set to the local MinIO component included in the Helm chart, which is suitable for debug purposes. For production however, it is recommended that this location should be on an object store owned by your organization (e.g. AWS S3).

Overriding the default large properties settings can be done by adding the following agent configuration in your custom-values.yml file. This setup covers the minimum settings you will need to set your large properties location to your organization’s objects store:

app-server:
  <other app-server custom values>
  configurationOverrides:
    - propertyKey: ldc.metadata.hdfs.large_properties.uri
      value: <large properties URI>
      component: __template_agent
    - propertyKey: ldc.metadata.hdfs.large_properties.attributes
      value:
        - fs.s3a.access.key=<access key>
        - fs.s3a.secret.key=<secret key>
        - fs.s3a.endpoint=<object store endpoint>
        - fs.s3a.path.style.access=true
        - fs.s3a.threads.max=20
        - fs.s3a.connection.maximum=200
      component: __template_agent
    - propertyKey: ldc.metadata.hdfs.large_properties.path
      value: <large properties path>
      component: __template_agent

Where:

• <access key>, <secret key> and <object store endpoint> refer to the connection details to your organization’s object store. This will be used by Data Catalog to connect and write to your object storage.
• <large properties URI> is the URI of your object store. For S3 bucket the URI can be set to one of two formats, depending what environment the agent is running on:
  • For most cases - s3a://<Bucket Name>
  • For a remote agent running on EMR - s3://<Bucket Name>
• <large properties path> refers to the location in the object store that metadata will be stored.

If you need to change a single agent’s large properties configuration, you can do this on the UI by going to Management and click on Configurations, and changing the following MISC values for your agent:

• Attributes for discovery cache metadata store
• URI for discovery cache metadata store
• Relative location for a large properties metadata store

The spark driver history location is where spark execution logs are written to. By default, the spark history server location is on the local MinIO component included in the Helm chart. Setting the new spark history server location to a S3 bucket can be done by overriding spark-history-server configuration your custom-values.yml file:

spark-history-server:
  <other spark history server configuration>
  historyServer:
    events:
      s3:
        createBucket: true
        # either use existingSecretName OR access keys
        existingSecretName: "<existing secret name>"
        accessKey: "<access key>"
        secretKey: "<secret key>"
        # aws s3 session token (optional)
        sessionToken: ""
        endpoint: "<endpoint>"
        bucket: <bucket name>
        eventLogPath: "<event log path>"
        # s3 event history file system log directory
        historyPath: "<history path>"

Where:

• <existing secret name> refers to a Kubernetes secret that holds the credentials to your S3 bucket (this would have been set up by your Kubernetes admin separately). Note that if you are using an existing secret name, accessKey and secretKey do not need to be set here.
• <access key> and <secret key> refer to the connection details to your organization’s S3 bucket. Note that if you are using keys to access your S3 bucket, you do not need to define existingSecretName.
• <endpoint> is the endpoint of the S3 or MinIO bucket. By default, this points to local MinIO accessible to the agent via http://<release name>-minio-bundled:9000.
• <bucket name> and <path> refers to the location in the object store’s where the jars will be placed.
• <event log path> and <history path> are locations on the bucket that logs are stored in. By default, these are both set to /events/.

Though Data Catalog comes with its own MongoDB, you can edit your custom-values.yml by providing the URI to an external MongoDB instance:

app-server:
  mongodbURI: <MongoDB URI>

Where the MongoDB URI will be in the form mongodb://<username>:<password>@ <mongodb host>.

Though Data Catalog comes with its own Keycloak component for authentication, it is possible for Data Catalog to point to your organization’s own Keycloak configuration by overriding Keycloak configuration in your custom-values.yml:

keycloak:
  authServerUrl: <auth server url>
  callbackUrl: <callback Url>
  realm: <realm>
  clientID: <client ID>  
  clientSecret: <client secret>
  authUser: <username>
  authPass: <password>
  # -- Mapping of OAuth profile fields (on the right) to those in LDC Application (on the left)
  userFields:
    id: sub
    email: email
    username: preferred_username
    firsName: given_name
    lastName: family_name

Where:

• <auth server url> is the (accessible) base URL of your Keycloak realm's authorization endpoint.
• <callback Url> is the URL to that Keycloak will redirect the user to after granting authentication.
• <realm> is the name of your Keycloak realm.
• <client ID> is a value that will match your Application Name, resource, or OAuth Client Name.
• <client secret> is the value of your OAuth client secret.
• <username> and <password> are the credentials of the user used for role syncing.

JDBC Jars are required by Data Catalog agents to access and process data sources. For local agent, custom-values.yml can be configured to pull additional seed JDBC jars in two different ways:

• s3 – During initialization, the agent pulls JDBC jars from a specified location (e.g. your organization’s S3 bucket)
• http – During initialization, the agent will use a provided list of download links to the jars agent requires. can pull jars via URL.

This is useful if local agent unexpectedly terminates, as a replacement of the agent will retrieve required jars during initialization.

By default, local agent will pull from JDBC jars from a bucket in the local MinIO component included in the Helm chart. In production, it is recommended that this location is set to an organization’s own object store, such as an AWS S3 or MinIO bucket. You can override your local agents jar file location by adding the following to your custom-values.yml helm chart.

agent:
  seedJDBC:
    sources:
      - s3
  # default values used in default secret
    s3:
      existingSecretName: "<secret name>"
      accessKey: <access key>
      secretKey: <secret key>
      bucket: <bucket name>
      path: <path>
      endpoint: "<endpoint>"

Where:

• <access key> and <secret key> refer to the connection details to your organization’s S3 or MinIO bucket.
• <endpoint> is the endpoint of the S3 or MinIO bucket. By default, this points to local MinIO accessible to the agent via http://<release name>-minio-bundled:9000).
• <bucket name> is the name of the S3 or MinIO bucket.
• <path> refers to the location in the S3 or MinIO bucket where the jars will be placed. By default, this is set to ext/jdbc.
• <secret name> refers to the name of the object store. By default, this is set to <release name>-minio-bundled.

For HTTP configuration, you can provide a list of download links to different jar files in your custom-values.yml:

agent:
  seedJDBC:
    sources:
      - http
    http:
      # array of links
      list:
      - <download link to a JDBC jar>
      - <download link to another JDBC jar>

Not to be confused with Spark History Server history server, this configuration is defined on the Agent's side and provides a path where the spark driver can upload additional jars for executors.

By default, this location is on the local MinIO component included in the helm chart, which can be used for debug purposes. For production however, it is recommended that this location should be set to an object store owned by your organization (e.g. AWS S3). This can be done by overriding agent configuration your custom-values.yml file:

agent:
  spark:
    jarUpload:
      endpoint: "<endpoint>"
      existingSecretName: "<secret name>"
      accessKey: <access key>
      secretKey: <secret key>
      secretToken:
      bucket: <bucket name>
      path: <path>

Where:

• <secret name> refers to the name of the object store. By default, this is set to <release name>-minio-bundled.
• <access key> and <secret key> refer to the connection details to your organization’s S3 or MinIO bucket.
• <endpoint> is the endpoint of the S3 or MinIO bucket. By default, this points to local MinIO accessible to the agent via http://<release name>-minio-bundled:9000.
• <bucket name> and <path> refers to the location in the object store’s where the jars will be placed.
• <path> refers to the location in the object store’s where the jars will be placed. By default, this is set to /cluster_jars.

Once you have made your edits, save the file, and run a helm upgrade using the updated custom-values.yml file:

Where <release name> and <namespace> refer to values that were set during the initial helm installation process.

If a server is reused to install Data Catalog, a resource conflict message may appear if the previous release has not been cleanly uninstalled (or fails):

Error: rendered manifests contain a resource that already exists.
Unable to continue with install: existing resource conflict:

If this error appears, manually delete any pre-existing resources:

kubectl delete all -l “app.kubernetes.io/instance=ldc” -l “release=ldc7”

For most cases, the provided helm uninstall command should suffice. However, if the uninstall is not completed successfully, you can run the command without enabling uninstall hooks:

helm uninstall <release name> --no-hooks -n <namespace>

If there are any orphaned or incomplete jobs, these can be found and deleted manually:

kubectl get jobs -o name | grep <release name>