This article documents the grace period provided by the virtual cluster activation key, when exceeding allowed number of nodes.

Resolution

When using a virtual cluster activation key, you are allowed to have an overage of cluster nodes up to double your allotted number of cluster nodes for a 2-hour grace period. Please note that cluster nodes exceeding the allowed overage (ie: those that are more than double your allotted number of nodes) will be deactivated immediately upon start up, and will show the activation error to any user accessing those nodes.

When the number of cluster nodes is exceeded into overage (up to double your allotted amount) , the cluster will start counting the 2-hour grace period and display grace period warnings in the console and for admin users in the UI, on all nodes.

When the grace period has ended, the overage cluster nodes (those up to double your allotted number) will start displaying the activation error page, as well. The cluster nodes that are within your allotted amount (not overage) will continue to be fully functional and display a warning, saying the grace period has ended.

Auto-Recovery

At the end of the grace period, the most recently started overage nodes will automatically receive shutdown commands one by one, so the cluster can return to an appropriate registered state.

During this time, some users will notice activation error messages if they are accessing nodes that have exceeded the allotted amount and grace period, but haven't been shutdown yet. These error messages will gradually resolve as the cluster automatically removes those nodes, one by one.

 

How Overage works with Virtual Cluster Keys

 

The time for the system to fully auto-recover itself is equal to the number of nodes exceeding the allowed amount multiplied by the node shutdown time.

Liferay DXP clustering functionality is based on the assumption of a single startup or shutdown process at a given time. If any other instances are started up or shut down at the same time, the cluster might reach an inconsistent state with unexpected errors. Please avoid triggering such operations during the recovery period.

Manual Recovery

System administrators that prefer to have full control over their cluster startup and shutdown processes can choose to disable the auto-recovery feature by setting this property in their portal-ext.properties file:

license.cluster.overload.node.auto.shut.down =  false

This property can only be configured through portal-ext.properties and has to be set before starting the cluster. It cannot be changed while the system is running.

When using this option, please keep in mind that nodes exceeding the allowed amount that are not shutdown will not be functional after the end of the grace period. They will show the activation error to any user accessing that node.

In this scenario, sys admins can shut down any cluster node, not only the most recently started ones. If an active node is shut down, the nodes exceeding the allowed amount will adjust shortly, as the activation status is communicated in the cluster. Deactivated or grace period nodes do not need to be restarted to become active. When an active node is shut down, the node exceeding the allowed amount with the earliest startup time will become active.