Skip to main content
Hitachi Vantara Lumada and Pentaho Documentation

Monitoring job activity

Parent article

Lumada Data Catalog maintains a history of job activity. You can view this history by navigating to Manage and selecting Job Activity. The Job Activity page displays all the job templates and individual job sequences that have been submitted by the users, and the status of those jobs. The jobs and their details are listed in the order in which they were submitted.

List  of jobs on the Job Activity page

FieldDescription
StartStart time for the job.
Sequence Name/Template NameName of the sequence or template triggering the job. If a job is triggered using a sequence, the sequence name displays, and the Template Name field is blank. If a job is triggered using a template, the template name displays, and the Sequence Name field is blank.
Asset NameName of the target resource of the job.
AgentName of the agent performing the job.
ElapsedElapsed time of the job.
Submitted ByThe user that triggered the job.
StatusThe job completion status.

Using the Job Activity page

You can view the details of a job by clicking its entry in the Job Activity page. The Job Info pane displays, detailing the sequence or sequences executed for that job.

The icon in the Status column indicates the status of the job. Even when the Status indicates Success, it is possible that the processing status for one of the resources may have failed. To verify the status of the resources, check for the counts of Success/Skipped/Incomplete in the Job Info pane for more details.

In the following example, a sequence or template has triggered a Profile Combo job. The Instance Steps section of the Job Info pane displays the execution details for all the steps involved: Format Discovery, Schema Discovery, and Profile.

The Job Activity page showing the Job Info pane

IconStatusMeaning
Submitted status iconSubmittedThe initial status of a job while it is being set up. This status transitions to In Progress.
In progress status iconIn ProgressIndicates that the job processes are running.
Success status iconSuccessIndicates that the job has completed successfully. For Profile Combo sequences the status switches between In Progress and Success/Failed for each of the sub jobs involved.
Success with warnings status iconSuccess with Warnings

Indicates that the job has completed successfully, but that parts of the job were unsuccessful.

Common reasons for this are:

  • There were permission issues of the resources on which the job is being performed.
  • There were conditions outside of the Data Catalog that were incorrect, such as Solr or Zookeeper were inaccessible.

When the job status is Success with Warnings, a WaterlineDataException or FileNotFound error may appear in the Data Catalog job logs. For more details, refer to the Job Info pane or to the job logs. The default log location is the var/log/waterlinedata directory.

Failed status iconFailedIndicates that the job has failed. Refer to the individual job logs for details.
Cancelling-cancelled status iconCanceling & CanceledThe Canceling status indicates a cancel action is in progress. The Canceled status appears when the job dependencies have been successfully freed and the cancel action has succeeded.
Incomplete status iconIncompleteThe count of resources within the data asset that could not finish discovery due to issues. The numbers indicate (total file skips + incompletes/total files).
Skipped status iconSkippedThe job was skipped. For example, the job may not have matched its resource permissions.

Using the Job Info pane

When you select a job on the Job Activity page, the Job Info pane displays information about that job in the Job Info section. The Instance Steps section displays information about each of the steps in the job. Click the down arrow next to a step to view the processing details of that step.

Job info detail

In addition to including the details captured on the Job Activity page, the Job Info pane has more details in the Job Info section and the Instance Steps sections.

Job Info section

The Job Info section contains the following fields:

FieldDescription
Instance IDThe internal reference id used by Data Catalog.
Asset TypeIdentifies whether the asset is a virtual folder, dataset, or data object.
Asset NameThe name of the asset.
Asset Path ListSpecifies the path of the resource or asset.
StartTime the job began.
ElapsedLength of time the job ran.

Instance Steps section

The Instance Steps section contains the following fields:

FieldDescription
StatusStatus of the step.
CommandThe internal command Data Catalog executes for the job. These commands can also be executed from the command line.
Spark execution IDThe Spark application ID for the current job instance.
Spark event logThe path to the Spark event log.
Total sizeTotal size of the asset processed.
SuccessCount of resources that were successfully processed.
SkippedCount of resources that were skipped. Directories are identified as file structures and are skipped while their contents are processed successfully. Resources that are identified as corrupt or in an unsupported format are skipped. Click the count links to view details.
IncompleteCount of resources that were incompletely processed. Clicking on the count links will list details.
Lineage InsertedCount of lineages inserted for the current lineage job.
Tag Associations InsertedCount of tag associations inserted for the current tag job.
Tag Associations RemovedCount of tag associations removed for the current tag job.
StartTime the step began.
EndTime the step ended.
Log File (button)Navigates to the Manage Logs menu and displays a snippet of the log file contents for this job. See Viewing Data Catalog logs for more details.
Download Log (button)Downloads the log for the step to your local machine.
Spark History (button)Opens the Spark history server and displays details for the instance.

The following is an example of a log file:Example snippet of log file contents for the selected job

Terminate a job instance

Perform the following steps to terminate a job instance after it has been submitted:

Procedure

  1. Navigate to Manage and select Job Activity.

    Job list showing the selected job properties link

  2. Select the instance (template or sequence) you want to terminate in the jobs list view.

  3. Click the number of selected properties hyperlink and select Terminate Instance.

    The instance is terminated. The status for terminated instances changes to Canceled.
    NoteBecause the termination of an instance is handled by the Spark engine, a terminated sequence may display a Success status in spite of the terminate action command from the user interface. This is more likely to occur with single sequence commands like format/schema/profile/tag when triggered on small sized resources.

Resubmit a job

You can resubmit jobs from the Job Activity page. Perform the following steps to resubmit a job:

Resubmit a job

Procedure

  1. Select the job you want to resubmit in the jobs list view.

  2. Click the selected properties hyperlink and select Resubmit job.

    The job is submitted with a new Instance ID.

Monitoring Spark jobs

When you run Data Catalog jobs, Data Catalog submits these jobs as spark context to the Apache Spark engine. Relevant Spark logging information is captured in the Data Catalog logs.

For example:

INFO  | 2020-07-07 07:31:54,369 | profile | ResourceExplorerRunner [main]  -    spark.yarn.historyServer.address = hdp265.ldc.com:18081
INFO  | 2020-07-07 07:31:54,369 | profile | ResourceExplorerRunner [main]  -    spark.driver.extraJavaOptions = -Dldc.log.dir=/var/log/ldc -Dldc.home=/opt/ldc/agent -Dlog4j.configuration=log4j-driver-client.xml -Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl -Djavax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Dldc.kerberos.keytab.file=ldcuser.keytab -Dldc.kerberos.principal=ldcuser@hitachivantara.com
INFO  | 2020-07-07 07:31:54,369 | profile | ResourceExplorerRunner [main]  -    spark.history.ui.port = 18081
INFO  | 2020-07-07 07:31:54,369 | profile | ResourceExplorerRunner [main]  -    spark.driver.extraClassPath = /opt/ldc/agent/lib/dependencies/commons-lang3-3.9.jar:/opt/ldc/agent/conf:/opt/ldc/agent/keytab:/opt/ldc/agent/ext/hive-serde-1.0.1.jar:/opt/ldc/agent/ext/mssql-jdbc-8.2.0.jre8.jar:/opt/ldc/agent/ext/postgresql-42.2.5.jar:/opt/ldc/agent/ext/ldc-hive-formats-2019.3.jar
INFO  | 2020-07-07 07:31:54,369 | profile | ResourceExplorerRunner [main]  -    spark.driver.extraLibraryPath = /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64
INFO  | 2020-07-07 07:31:54,369 | profile | ResourceExplorerRunner [main]  -    spark.history.kerberos.principal = spark-hdp265@hitachivantara.com

Alternatively, you can monitor Apache Spark jobs at one of the following sites:

  • Cloudera's Apache Spark History Server: http://<cluster IP address>:8888/jobbrowser
  • Ambari's Apache Spark History Server: http://<cluster IP address>:19888/jobhistory

You may need to additionally specify the Data Catalog service user or, if the service user has a corresponding account, sign in using that user account.

Collecting debugging information

Data Catalog features multiple sources of monitoring information. If you encounter a problem, collect the following information for Data Catalog support.

See Setting logging properties for more information about each source in the following list.

  • Job messages

    Data Catalog generates console output for jobs run at the command prompt. If the job encounters problems, review the job output for clues to the problem. These messages appear on the console and are collected in log files with the debug logging level: /var/log/ldc/ldc-jobs.log.

  • Spark job messages

    Many Data Catalog jobs trigger Spark operations. These jobs produce output that is accessible on the cluster job history server and also through Hue’s job browser or the Ambari Spark job history.

    You can open the Spark job log that corresponds to the profiling operation for a given file or table by opening that resource in the browser and following the link in the profile status.

  • Application server messages

    The embedded Jetty application server produces output corresponding to user interactions with the browser application, including the installation process. These messages appear on the console and are collected in a log file: /var/log/ldc/ldc-ui.log

    Use the tail command to see the most recent entries in the log:

    $ tail -f /var/log/ldc/ldc-ui.log