Runtime Configuration

Flux Engineers

January 09, 2021 07:18

There are two configuration components for the Flux engine:

Static. Static configuration options are described in greater detail in the Engine Configuration chapter.
Runtime. Runtime configuration options typically determine how flow charts run (for example, limiting the number of flow charts that can run at once) or contain data that the flow charts will use while running. Unlike the static configuration, the runtime configuration is dynamically reloaded, so runtime configuration settings can change while the engine is running. Runtime configuration options are applied to nodes, or branches, on the flow chart namespace tree, allowing you to define runtime configuration properties on a per-namespace level.

Every engine is controlled by both its static (non-runtime) configuration and its runtime configuration.

The runtime configuration is option. To enable the runtime configuration, you must first specify the configuration file location, then set the properties you would like to use.

Setting the Runtime Configuration File

To use a runtime configuration file, you can simply add the following property to your engine configuration:

RUNTIME_CONFIGURATION_FILE=/path/to/myfile.config

Refreshing the Runtime Configuration File

If you make changes to the runtime configuration file, Flux can automatically and dynamically pick up your changes while the engine is running (without requiring an engine restart). This refresh is performed at a specific interval, which you can adjust by setting the RUNTIME_CONFIGURATION_FILE_REFRESH_FREQUENCY like so:

RUNTIME_CONFIGURATION_FILE_REFRESH_FREQUENCY=+30s

This property is a time expression. The default value is +m (every one minute).

Setting Properties in the Runtime Configuration

The Flux runtime configuration is a standard Java properties file. This means that each line of the file is expected to contain a single key / value pair.

In the runtime configuration file, the key is expected to be a namespace corresponding with your flow chart namespace (see #Tree of Configuration Properties below for more information) plus the name of the property you want to set. The value of each pair is the value should be the intended value of the property you are setting.

There are two types of properties available:

Runtime Configuration Property. These properties configure how flow charts are executed on the engine. Runtime configuration properties may include concurrency throttle settings, priority values, FIFO settings, and more. See the #List of Runtime Configuration Properties for a complete list of available properties of this type.
Runtime Variables. These are custom variables that you can use to provide data to your flow charts. Typically, this would be used when a single value (for example, a username or password) is used in several flow charts. Using runtime variables allows to specify variable values in a single location, rather than re-entering them across several flow charts. You can use variable substitution or runtime data mapping to access the runtime variable values inside a flow chart.

Runtime configuration properties are set like so:

/namespace/<PROPERTY TYPE>=<value>

Runtime variables, similarly, are set like:

/namespace/<VARIABLE NAME>=<value>

Runtime configuration and variable properties can be set on any namespace, including the root ("/"). These settings are applied to all flow charts on the namespace, as as well as all flow charts on child namespaces.

A sample runtime configuration might look like:

/MAIL_SERVER_VARIABLE=mymailserver.com
/CONCURRENCY_THROTTLE=8
/namespace/CONCURRENCY_THROTTLE=5

Runtime configuration namespaces

Runtime configuration properties can only be applied to a namespace, not directly to a particular flow chart or flow chart name. This means, for example, that a flow chart with the full namespace (including flow chart name) of:

/MyNamespace/MySubNamespace/MyFlowChartName

Will pick up any properties set in the runtime configuration for namespaces:

/MyNamespace

And:

/MyNamespace/MySubNamespace

But a property set on the full name of:

/MyNamespace/MySubNamespace/MyFlowChartName

Will be ignored. Keep this in mind when naming your flow charts and designing the runtime configuration.

List of Runtime Configuration Properties

The following properties can be set in the runtime configuration. All of the properties listed can be specified across the entire engine (by setting them on the root namespace), or assigned to an individual namespace.

Property Name	Description
CONCURRENCY_THROTTLE	The total number of flow charts that can run concurrently within this namespace. For example, a concurrency throttle of 5 would mean that 5 flow charts in this namespace should be allowed to run at once. Note that there is one exception to this: if a flow chart contains a split (or, in general, any concurrently executing actions), each action will count toward the total concurrency. For example, if a flow chart contains 5 actions running at once, and the namespace has a concurrency throttle of 5, that flow chart will use all of the available concurrency slots and no other flow charts will be allowed to run in the same namespace until the actions complete. The concurrency throttle setting allows you to conserve limited resources (like processing power and I/O bandwidth) by preventing an unbounded number of jobs from running at once. You can also set a concurrency throttle of 0 to prevent a certain namespace from running at all. For example, if you have flow charts which require specific resources available on the machine, you could use concurrency throttles to limit which machines those flow charts are allowed to run on.
CONCURRENCY_THROTTLE_CLUSTER_WIDE	Identical to the CONCURRENCY_THROTTLE setting, except that this property specified the total number of flow charts in this namespace that can be allowed to run across the entire cluster. For example, if engines A and B are running in a cluster, the CONCURRENCY_THROTTLE_CLUSTER_WIDE for each engine is set to 8, engine A is running 5 flow charts, and engine B is running 3 flow charts, then neither engine will accept any new flow charts until the concurrency slots are available again. This property overrides the individual concurrency throttle for each engine – an engine will not accept a flow chart if the cluster wide throttle for that flow chart's namespace is full, even if there is room in the engine's CONCURRENCY_THROTTLE settings for that namespace.
DEFAULT_FLOW_CHART_ERROR_HANDLER	The error handler that flow charts in this namespace should use. This property should be set as a path to the error handler file. The path can be absolute, like "/path/to/myfile.ffc", or it can be relative to the Java home directory (that is, the directly where the JVM was started, usually the directory containing the script to launch the Flux engine). A relative path may look like "folder/myfile.ffc".
FIFO_SCHEDULING_ENABLED	Normally, flow charts that are eligible for execution are scheduled for execution based on their priority and their "next execution date". Sometimes, however, it is desirable to schedule flow charts to run in a first-in-first-out (FIFO) order. For example, suppose 10,000 trucking manifests need to be generated, and the manifest flow charts that were scheduled first should be, generally speaking, processed first. Flux normally processes such flow charts in this manner by default. However, the existence of a trigger or an error handler in a flow chart can cause a flow chart to "go to the back of the queue". In these cases, it may be that the trucking manifest flow chart that was scheduled first may end up running behind the ones that were originally submitted afterwards. To ensure that flow charts are executed in the order in which they were originally submitted to an engine, you can use Flux's FIFO Scheduling feature. When enabled, this feature ensures that flow charts are scheduled for execution based on the timestamp at which they were originally submitted to an engine (except for flow chart priorities, which take precedence over FIFO Scheduling). If a flow chart is paused, it is not considered in the FIFO Scheduling algorithm. Continuing the trucking manifest example, if an engine's concurrency throttle is 10 and there are 100 trucking manifest flow charts submitted to an engine, up to 10 manifest flow charts may execute simultaneously. FIFO Scheduling ensures that those 10 running flow charts were submitted to the engine earlier than the remaining 90 flow charts. To achieve a true queue-like behavior, simply set the concurrency throttle to one. That configuration ensures that one flow chart must either run to completion or reach a trigger before the net flow can execute. FIFO scheduling is disabled by default.
INTERNAL_LOGGER_DEBUG_ENABLED	If this property is set to true, action and trigger properties and variables for flow charts in this namespace will be logged in greater detail to the log file and audit trail. As the name implies, this property can only be used if the internal logger is enabled.
LISTENER_CLASSPATH	Specifies the listener class path for flow charts in this namespace. This property specifies the listener classpath for flow charts, which is used to load listener classes in Java Actions and Dynamic Java Actions.
PRIORITY	Specifies the priority for flow charts in this namespace. Priorities are used to give precedence to certain flow charts when there are more flow charts available to run than concurrency slots open to run them.
RUN_AS_USER	The username of a user whose permissions will be used to run this flow chart. This can only be used when using the built-in Flux security mechanism.

Tree of Configuration Properties

The runtime configuration is set up as a tree of nodes (following the normal tree structure). Runtime configuration settings are applied on a per-namespace level, so the runtime configuration tree will mimic the flow chart namespace tree. A property set on a branch in the runtime configuration tree is applied to the branch of the same name in the flow chart namespace tree.

Runtime configuration namespaces are not case-sensitive. For example, the namespace "/MyNamespace" is equivalent to setting "/mynamespace" in the runtime configuration.

Each configuration property consists of a key and a value. The key must be a string. The value must be a persistent variable.

Although the runtime configuration tree can contain any kind of property, it includes pre-defined properties: concurrency throttle, concurrency throttle cluster-wide, default flow chart error handler, and priority.

Flow chart names are hierarchical. They form a tree of jobs. For example, the two flow charts, "/lightweights/job 1" and "/heavyweights/job 2", form a tree with two branches off the root, "/lightweights" and "/heavyweights".

You can annotate this tree of jobs using properties in the runtime configuration. For example, to set a concurrency throttle for heavyweight jobs, set an appropriate concurrency throttle in the "/heavyweights" branch of the runtime configuration tree. To set a default flow chart error handler for lightweight jobs, set an appropriate error handler in the "/lightweights" branch of the runtime configuration tree.

Properties set on a child namespace will override any properties set on a parent. For example, if a flow chart runs in the namespace "/parent/child/", it will first check the namespace "/parent/child/" for any runtime properties. Properties for the "/parent" namespace would only be applied if they were not set in "/parent/child".

In summary, the tree of configuration properties mirrors the tree of jobs. A job’s behavior can be further configured based on the branches in the runtime configuration tree that correspond to that job’s fully qualified name.

Runtime Configuration Trees on Externally Secured Engines

If your engine has external security enabled, the runtime configuration tree must contain the name of the group for the properties you are trying to set, as well as the namespaces you are setting them on. The group name should be included at the base of the configuration tree, preceding the namespace.

When using an external security provider (such as LDAP), all users and flow charts are automatically assigned to the "default" group. Therefore, your runtime configuration might look something like:

/default/mynamespace/CONCURRENCY_THROTTLE=2

Repeat this as necessary for each namespace and property combination you want to set.

Runtime Configuration Properties for Namespaces that contain a space

If your namespace contains a space character, like:

/Folder A/MyFlowChart

You will need to make sure the space is escaped in your runtime configuration by adding a backslash ('\') character before the space in the configuration. For example, to set a runtime configuration property for a flow chart with the name and namespace above, you could add the following line to your runtime configuration file:

/Folder\ A/CONCURRENCY_THROTTLE=0

Memory Usage

It is assumed that your entire runtime configuration tree can be stored in memory. A very large runtime configuration can cause memory problems in Flux, so take care to ensure that you are not setting any unnecessary or extraneous data in the tree.

Storing Encrypted Passwords in the Runtime Configuration

If you have a password value that is used in several flow charts across the engine, it often makes sense to store the password in one central location where all of the flow charts can easily access it.

In this case, you would normally not want to store the password in plain text in the runtime configuration either (where any user with access to the runtime configuration file or the logs might access it). To help with this, Flux allows you to store an encrypted value in the runtime configuration, protecting your password values and ensuring that they are not available in plain text anywhere on the system.

To store an encrypted password in the runtime configuration and access it within your flow charts, follow these steps:

Follow the Command Line Interface for generating an encrypted password. This will create an encrypted password which should look something like Gc4f0oRkPJ4=.
Edit your runtime configuration to store the encrypted password. The encrypted password doesn't require any special syntax and can be set like any other variable:

/MyEncryptedPassword=Gc4f0oRkPJ4=
Use runtime substitution to access the value in the password field on your action:

${runtime MyEncryptedPassword}

That's it! This technique can be used for any action in Flux that requires a password, including file actions and triggers that use a password for authentication to a remote host.

The Runtime Configuration File and Java API

When using the runtime configuration file, it is automatically refreshed from the file system to pick up any changes that might be made. For this reason, it is important that you do not make any changes to the runtime configuration using the API if you are using a runtime configuration file (the engine has no way of knowing whether the file or API changes are the "correct" version, so the automatic file refresh assumes that it is correct and will overwrite any changes you have made in the API).

If you are using the runtime configuration file, any changes to the runtime configuration should be performed in the file alone. If you are using the API, the runtime configuration should be managed entirely that way, avoiding the runtime configuration file altogether.