Skip to main content
Version Pentaho
Hitachi Vantara Lumada and Pentaho Documentation

Specify Additional Spark Properties

  1. Last updated
  2. Save as PDF

You can define additional Spark properties in two ways:

  • Using the application.properties file
  • Using run modification parameters within a transformation

Any property that starts with spark. is passed directly to the Spark configuration.

Use the Properties File

Within the application.properties file, you may add any number of Spark configuration properties, as detailed at: https://spark.apache.org/docs/latest/configuration.html 

Use Run Modification Parameters

You may also define any additional Spark configuration properties as run modification parameters within a transformation.

Learn more

Order of Processing

Because Spark properties can be set in multiple ways, it is important to understand the following order of processing:

  1. Spark properties set on the Pentaho Server apply to all users and all transformations. 
  2. Spark properties specified within a .ktr apply to the specific user who runs the transformation. For example, if a user wants to change the memory size, they can embed the appropriate Spark parameter in the .ktr so that it executes when they run the transformation.

If an identical property is also set on the Pentaho Server, the user's.ktr takes precedence.