Prerequisites and follow-up actions for upgrading the Lumada Data Catalog solution in LDOS

Procedure

1. Retrieve the Postgres user secret from the cluster by running the following command:

   kubectl get secrets -n <name-space>The Postgres user secret appears.
   NoteThe format of the secret is postgres.t0-catalog-pg-cluster.credentials.postgresql.acid.zalan.do where t0-catalog-pg-cluster is the name of the cluster.

2. Create a persistent volume claim (PVC) file in the hitachi-solutions namespace to back up the data. Save it as postgres-backup-restore-pvc.yaml as in the following example:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
   
     name: postgres-backup-restore-volume
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 20Gi

3. Create a database backup job and define it as postgres-backup-job.yaml. Replace the variables <postgres-master-pod-svc-name>, <postgres-user-secret>, and <ldc-data-base> with the applicable values for your system:

   NoteBe sure to use the Postgres secret, not the ldcuser secret.

   The following sample is an example of the postgres-backup-job.yaml file.

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: postgres-backup
   spec:
     template:
       metadata:
         name: backup-pg-data
       spec:
         containers:
           - name: postgres-backup
             image: postgres:12
             imagePullPolicy: Always
             env:
               - name: POSTGRES_HOST
                 value: <postgres-master-pod-svc-name>
               - name: POSTGRES_USER
                 valueFrom:
                   secretKeyRef:
                     name: <postgres-user-secret>
                     key: username
               - name: POSTGRES_PASSWORD
                 valueFrom:
                   secretKeyRef:
                     name: <postgres-user-secret>
                     key: password
               - name: POSTGRES_DB
                 value: <ldc-data-base>
               - name: BACKUP_FILE
                 value: /var/backups/postgres-data.tar
             volumeMounts:
               - mountPath: /var/backups
                 name: postgres-backup-pvc
             command:
               - sh
               - -c
               - |
                 #!/usr/bin/env bash
                 export PGPASSWORD=$POSTGRES_PASSWORD
                 
                 echo "Postgres data backup: $BACKUP_FILE"
                 until pg_isready -h $POSTGRES_HOST -U $POSTGRES_USER -d $POSTGRES_DB; do sleep 2; done;
                 pg_dump -h $POSTGRES_HOST -U $POSTGRES_USER -Ft $POSTGRES_DB > $BACKUP_FILE
                 echo 'Backup finished!'
         restartPolicy: Never
         volumes:
           - name: postgres-backup-pvc
             persistentVolumeClaim: 
               claimName: postgres-backup-restore-volume

4. Deploy the PVC file using the following command:

   kubectl apply -f postgres-backup-restore-pvc.yaml

5. Run the backup job using the following command:

   kubectl apply -f postgres-backup-job.yaml

Results

Parent Topic

• Prerequisites and follow-up actions for upgrading the Lumada Data Catalog solution in LDOS

Before you begin

Procedure

1. Enable the backup functions in LDC by adding the following code to the custom-values.yaml file in the LDOS user interface:

   solr:
     cloud:
       backupRestoreData:
         enabled: true

   NoteSee Configuring solutions.

2. Create a persistent volume claim file in the hitachi-solutions namespace to back up the Solr data. Save it as solr-backup-pvc.yaml using the following code:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: solr-backup-persistence
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 10Gi

3. Create the custom resource definition (CRD) file to back up the Solr data into a tgz file, as in the following sample:

   apiVersion: solr.bloomberg.com/v1beta1
   kind: SolrBackup
   metadata:
     name: solrbackup
   spec:
     persistence:
       volume:
         source:
           persistentVolumeClaim:
             claimName: solr-backup-persistence
     solrCloud: {{ .Release.Name }}
     collections:
       - {{ collection-name }}

   1. Replace {{ .Release.Name }} with the name of the release where your SolrCloud is running.

   2. Replace {{ collection-name }} with the name of the Solr collection that you want to back up.

4. Save the CRD file as solr-backup.yaml.

5. Run the CRD file using the following command: kubectl apply -f solr-backup.yaml -n <namespace>.

Results

Parent Topic

• Prerequisites and follow-up actions for upgrading the Lumada Data Catalog solution in LDOS

Procedure

1. Navigate to the s3a://ldc/home/ldcuser bucket.

2. Download the ldc_hdfs_metadata.zip file and store it in a location where you can access it to restore the backup.

Parent Topic

• Prerequisites and follow-up actions for upgrading the Lumada Data Catalog solution in LDOS

Procedure

1. Create a database restore job using the following code and save it as postgres-restore-job.yaml.

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: postgres-restore
   spec:
     template:
       metadata:
         name: restore-pg-data
       spec:
         containers:
           - name: postgres-restore
             image: postgres:12
             imagePullPolicy: Always
             env:
               - name: POSTGRES_HOST
                 value: <postgres-master-pod-svc-name>
               - name: POSTGRES_DB
                 value: <ldc-data-base>
               - name: POSTGRES_USER
                 valueFrom:
                   secretKeyRef:
                     name: <postgres-user-secret>
                     key: username
               - name: POSTGRES_PASSWORD
                 valueFrom:
                   secretKeyRef:
                     name: <postgres-user-secret>
                     key: password
               - name: BACKUP_FILE
                 value: /var/backups/postgres-data.tar
             volumeMounts:
               - mountPath: /var/backups
                 name: postgres-restore-pvc
             command:
               - sh
               - -c
               - |
                 #!/usr/bin/env bash
                 export PGPASSWORD=$POSTGRES_PASSWORD
   
                 echo "Restoring the postgres data from file $BACKUP_FILE"
                 until pg_isready -h $POSTGRES_HOST -p 5432 -U $POSTGRES_USER -d $POSTGRES_DB; do sleep 2; done;
                 pg_restore -U $POSTGRES_USER -h $POSTGRES_HOST -Ft --clean -d $POSTGRES_DB < $BACKUP_FILE
                 echo 'Postgres data restored!'
         restartPolicy: Never
         volumes:
           - name: postgres-restore-pvc
             persistentVolumeClaim:
               claimName: postgres-backup-restore-volume

2. In the code, replace the following variables with the values for your environment:

   • <postgres-master-pod-svc-name>
   • <postgres-user-secret>
   • <ldc-data-base>
   NoteYou must use the Postgres user secret credentials that you retrieved in the backup process. If you use the ldcuser secret, then the data restoration fails.

3. Run the database restore job.

   NoteIf the Postgres database has been deleted, you can re-create the database with your restore job by using the --create option with the pg_restore command. For example, pg_restore -U $POSTGRES_USER -h $POSTGRES_HOST -Ft --clean --create -d $POSTGRES_DB < $BACKUP_FILE .

Results

Parent Topic

• Prerequisites and follow-up actions for upgrading the Lumada Data Catalog solution in LDOS

Procedure

1. Create a restore job using the following code.

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: solr-collection-restore
   spec:
     template:
       metadata:
         name: collection-restore
       spec:
         restartPolicy: Never
         containers:
           - name: restore-collection-job
             image: registry_name/lumada-data-catalog/solr-collection-manage-hook:8.4.1-6.1.1
             volumeMounts:
               - mountPath: /opt/solr/backup-mount
                 name: solr-backup-persistence-volume
               - mountPath: /var/solr/data/backup-restore
                 name: solr-backup-restore-volume
                 subPath: cloud/t0
             env:
               - name: "SOLR_COLLECTION"
                 value: {{ collection-name }}
               - name: "SOLR_HTTP_URL"
                 value: {{ solr-http-url }}
               - name: "JOB_MODE"
                 value: "RESTORE_COLLECTION"
               - name: "SOLR_BACKUP_NAME"
                 value: "solrbackup"
               - name: "LDC_COLLECTION_HOOK_DEBUG"
                 value: "true"
         volumes:
           - name: solr-backup-persistence-volume
             persistentVolumeClaim:
               claimName: {{ backup-persistence-name }}
           - name: solr-backup-restore-volume
             persistentVolumeClaim:
               claimName: {{ restore-pvc-name }}

2. In the code, replace the following values with the values for your environment:

   • registry_name

   • {{ collection-name }}

     The collection name where you want to restore Solr

   • {{ solr-http-url }}

     The SolrCloud URL. You can use the SolrCloud common service name here. For example: http://t0-solrcloud-common

   • {{ backup-persistence-name }}

     The name of the PVC created in the Solr platform backup topic (solr-backup-persistence.yaml).

   • {{ restore-pvc-name }}

     The name of the PVC created in the Solr platform backup (custom-values.yaml) . This PVC was created during the catalog installation. The format is {{.Release.Name }}--solr-backup-restore-volume-claim

3. Save the edited code as the job file solr-collection-restore.yaml.

4. Run the solr-collection-restore.yaml job by entering the following command :kubectl apply -f solr-collection-restore.yaml -n hitachi-solutions.

Parent Topic

• Prerequisites and follow-up actions for upgrading the Lumada Data Catalog solution in LDOS

Procedure

1. Remove the data in this path: s3a://ldc/home/ldcuser.

2. Extract the ldc_hdfs_metadata.zip file from the location where you stored the discovery cache backup.

3. Upload the extracted data to s3a://ldc/home/ldcuser.

Results

Parent Topic

• Prerequisites and follow-up actions for upgrading the Lumada Data Catalog solution in LDOS