Payara Server Upgrade Tool

This tool can be used to upgrade an existing Payara Server installation to a newer version. The Upgrade Tool is bundled in Payara Server versions starting from release 5.24.0 and is maintained as a separate JAR artifact under the fish.payara.extras: payara-upgrade-tool GAV coordinates.

The upgrade tool can be manually installed to Payara Platform 5.20.0 and newer if needed. To do so, download the latest 1.x JAR release from Payara Nexus using this link and then drop it into your Payara Server installation under the payara5/glassfish/lib/asadmin folder, replacing an older version of the artifact in case it exists. The 2.x releases of the upgrade tool are for use with Payara Server 6 releases.

Upgrading to Payara Server 6

It is possible to upgrade a Payara Server 5.x domain to version 6.x. The process should not be any different as to upgrading between different Payara 5 minor versions, though with the following caveats:

The upgrade tool only supports upgrading to Payara Server 6 versions 6.x.y or higher.
When performing an upgrade from Payara 5 to Payara 6, make sure that you’re using JDK 11 as this is the minimum JDK version required to run Payara Server 6.

The upgrade tool will not adjust the default-web.xml configuration file (located in ${as-install}/config) for each domain, so the following changes have are required:

A <servlet-class> entry of org.apache.jasper.servlet.JspServlet must be converted to org.glassfish.wasp.servlet.JspServlet

The <system-jar-includes> initialization parameter of the jsp servlet has had a change in value between Payara Server 5 and Payara Server 6, reflecting the change in modules from Jakarta EE 8 to Jakarta EE 10.

This is how the parameter looks on a standard Payara Server 6.4.0 domain configuration:

<init-param>
  <param-name>system-jar-includes</param-name>
  <param-value>
    /lib/
    \lib\
    expressly.jar
    jakarta.servlet-api.jar
    jakarta.servlet.jsp-api.jar
    jakarta.servlet.jsp.jstl-api.jar
    jakarta.servlet.jsp.jstl.jar
    jakarta.jms-api.jar
    jakarta.faces.jar
    wasp.jar
    jspcaching-connector.jar
    web-glue.jar
    hibernate-validator.jar
    jakarta.ejb-api.jar
    jakarta.enterprise.deploy-api.jar
    jakarta.activation-api.jar
    angus-activation.jar
    jakarta.mail-api.jar
    angus-mail.jar
    jakarta.persistence.jar
    jakarta.resource-api.jar
    jakarta.security.auth.message-api.jar
    jakarta.security.jacc-api.jar
    jakarta.transaction-api.jar
    webservices-osgi.jar
    weld-osgi-bundle.jar
    jersey-mvc-jsp.jar
  </param-value>
</init-param>

You’ll have to adjust the value of this parameter by copying the current values of a clean Payara Server 6.4.0 or newer domain after the upgrade tool completes the process.

It is recommended that users should run the latest version of Payara Server 5.x before attempting to upgrade to version 6.x.

Asadmin CLI Commands

The following are the Asadmin CLI commands available to the upgrade tool.

Upgrade Server Command

Usage: asadmin> upgrade-server
Aim: This command can be used to upgrade Payara Server to the specified distribution along with all SSH nodes in the domain.xml configuration file.

The command will also back up the old version and domains in the default location (payara5/glassfish/domains) or the location specified using the --domaindir option to allow for rolling back.

Command Options

Option Type Description Default Mandatory

Option	Type	Description	Default	Mandatory
`--username`	String	The username for the Payara Nexus server.		Yes, unless `--usedownloaded` is specified
`--version`	String	Specifies the version number of the new version of Payara to use.		Yes, unless `--usedownloaded` is specified
`--distribution`	Enum	Specifies the distribution of Payara Server to download. This can payara, payara-web, payara-ml, or payara-web-ml.	payara	No
`--stage`	Boolean	Determines if the upgrade should be installed in-place, or staged into "x.new" directories. Scripts are provided that can be used to apply and rollback the staged install. This option is automatically enabled when running the command on Windows.	False on Unix, True on Windows	No
`--domaindir`	String	The directory containing the domains. The domains in this directory are backed up, and their config is used to determine the nodes which will also be upgraded.	`${as-install}/domains`	No
`--usedownloaded`	String	The path to the local Payara Server zip archive to use for upgrading instead of downloading from the Payara Enterprise repository. When this parameter is specified, the `--username`, `--password`, and `--version` parameters are not required.		No
`--nodes`	String	Specifies the nodes to upgrade if you want to selectively upgrade certain nodes. Provide as a comma-separated list of node names.	All nodes in the cluster	No

--username

String

The username for the Payara Nexus server.

Yes, unless --usedownloaded is specified

--version

String

Specifies the version number of the new version of Payara to use.

Yes, unless --usedownloaded is specified

--distribution

Enum

Specifies the distribution of Payara Server to download. This can payara, payara-web, payara-ml, or payara-web-ml.

payara

--stage

Boolean

Determines if the upgrade should be installed in-place, or staged into "x.new" directories. Scripts are provided that can be used to apply and rollback the staged install. This option is automatically enabled when running the command on Windows.

False on Unix, True on Windows

--domaindir

String

The directory containing the domains. The domains in this directory are backed up, and their config is used to determine the nodes which will also be upgraded.

${as-install}/domains

--usedownloaded

String

The path to the local Payara Server zip archive to use for upgrading instead of downloading from the Payara Enterprise repository. When this parameter is specified, the --username, --password, and --version parameters are not required.

--nodes

String

Specifies the nodes to upgrade if you want to selectively upgrade certain nodes. Provide as a comma-separated list of node names.

All nodes in the cluster

Example

This example upgrades a Payara Web distribution to version 5.24.1

asadmin> upgrade-server --username example-user --distribution payara-web --version 5.24.1

Rollback Server Command

Usage: asadmin> rollback-server
Aim: This command can be used to rollback Payara Server to the point before the upgrade-server command was run, restoring the most recent backup of the domain (expected to be the backup created during execution of the upgrade-server command).

This command is not supported on Windows OS, please use the rollbackUpgrade.bat script instead.

Command Options

Option Type Description Default Mandatory

Option	Type	Description	Default	Mandatory
`--domaindir`	String	The directory containing the domains. The domains in this directory are backed up, and their config is used to determine the nodes which will also be upgraded.	$`{as-install}/domains`	No
`--nodes`	String	Specifies the nodes to rollback if you want to selectively rollback certain nodes. Provide as a comma-separated list of node names.	All nodes in the cluster	No

--domaindir

String

The directory containing the domains. The domains in this directory are backed up, and their config is used to determine the nodes which will also be upgraded.

${as-install}/domains

--nodes

String

Specifies the nodes to rollback if you want to selectively rollback certain nodes. Provide as a comma-separated list of node names.

All nodes in the cluster

Staged Upgrades

When the upgrade-server command is either used on Windows or with the --stage option enabled, the new server files are installed next to the current installation in various .new directories (e.g. payara5/glassfish/bin.new). The following helper scripts are available to interact with staged upgrades.

Apply Staged Upgrade Script

Usage: > ./payara5/glassfish/bin/applyStagedUpgrade
Aim: This script is used to apply an upgrade staged using the upgrade-server command. It will move the current installation into .old directories, and the staged .new installation into the expected "current" location. It will then upgrade the nodes of the domains in the default domain dir, or the domains in the directory provided using --domaindir

Command Options

Option Type Description Default Mandatory

Option	Type	Description	Default	Mandatory
`--domaindir`	String	The directory containing the domains. The config of the domains in this directory are used to determine the nodes which will also be upgraded.	${as-install}/domains	No
`--nodes`	String	Specifies the nodes to upgrade if you want to selectively upgrade certain nodes. Provide as a comma-separated list of node names.	All nodes in the cluster	No

--domaindir

String

The directory containing the domains. The config of the domains in this directory are used to determine the nodes which will also be upgraded.

${as-install}/domains

--nodes

String

Specifies the nodes to upgrade if you want to selectively upgrade certain nodes. Provide as a comma-separated list of node names.

All nodes in the cluster

Rollback Upgrade Script

Usage: > ./payara5/glassfish/bin/rollbackUpgrade
Aim: This script is used to rollback a server upgrade applied using the applyStagedUpgrade script. It will move the .old installation back into the expected "current" location, and the applied upgrade back into .new directories. It will then rollback the nodes of the domains in the default domain dir, or the domains in the directory provided using --domaindir

Command Options

Option Type Description Default Mandatory

Option	Type	Description	Default	Mandatory
`--domaindir`	String	The directory containing the domains. The config of the domains in this directory are used to determine the nodes which will also be rolled back.	`${as-install}/domains`	No
`--nodes`	String	Specifies the nodes to rollback if you want to selectively rollback certain nodes. Provide as a comma-separated list of node names.	All nodes in the cluster	No

--domaindir

String

The directory containing the domains. The config of the domains in this directory are used to determine the nodes which will also be rolled back.

${as-install}/domains

--nodes

String

Specifies the nodes to rollback if you want to selectively rollback certain nodes. Provide as a comma-separated list of node names.

All nodes in the cluster

Cleanup Upgrade Script

Usage: > ./payara5/glassfish/bin/cleanupUpgrade
Aim: This script is used to clean up any leftovers from a staged upgrade: any .old folders and any .new folders will be deleted.

Use of this script will prevent you from rolling back or applying a staged upgrade.

This script cannot be used to clean up upgrades from releases prior to 5.26.0, as it uses a config file to determine the directories to delete which is only generated by the upgrade-server command from release 5.26.0 onwards.

Configure Logging Levels

The upgrade tool commands and helper scripts will print a set of minimum details of the operations executed (upgrade, staging, rollback). For troubleshooting scenarios, or if wanting to review in detail all executed actions, the following 2 environment variables are available to control the level of logging done by the Upgrade tool:

AS_DEBUG: Set to true to configure the Upgrade Tool’s logging level to FINER.
AS_TRACE: Set to true to configure the Upgrade Tool’s logging level to FINESET.

These variables can also be configured as system properties in the Asadmin CLI script file located in the {as-install}/bin folder like this:

AS_INSTALL=`dirname "$0"`/../glassfish
AS_INSTALL_LIB="$AS_INSTALL/lib"
. "${AS_INSTALL}/config/asenv.conf"
JAVA=java
#Depends upon Java from ../config/asenv.conf
if [ ${AS_JAVA} ]; then
    JAVA=${AS_JAVA}/bin/java
fi

exec "$JAVA" -DAS_DEBUG=true  -XX:+IgnoreUnrecognizedVMOptions -jar "$AS_INSTALL_LIB/client/appserver-cli.jar" "$@"

Remember to turn off these logger level settings after executing a server upgrade, as this setting will affect all future executions of any Asadmin CLI commands, which will cause them to print out more information than usual.

Zero Downtime upgrade

This feature requires payara-upgrade-tool.jar from Payara v5.83.0 or v6.34.0 and above.

By default, upgrading Payara requires that the domain and all SSH nodes are stopped. This feature allows the upgrading of individual SSH nodes, eliminating downtime. Only upgrades in the same major version are supported 5.x to 5.x, 6.x to 6.x. Upgrades between 5.x and 6.x are not supported.

Steps to upgrade:

stop the DAS

asadmin> stop-domain

upgrade the domain

asadmin> upgrade-server --username example-user --distribution web --version 5.24.1 --nodes localhost-domain

Stop one SSH node to upgrade

asadmin> stop-node-ssh node1

Upgrade one SSH node

asadmin> upgrade-server --username example-user --distribution web --version 5.24.1 --nodes node1

Start the upgraded SSH node

asadmin> start-node-ssh node1

Repeat the last 3 steps with the remaining SSH nodes to upgrade. Start the DAS

asadmin> start-domain

the upgrade can cause issues if the nodes are forming a cluster with Hazelcast. When 2 versions of Payara use different versions of Hazelcast, the upgraded node will fail to start as long as nodes using the older versions are running. If the upgraded node/DAS fails to start, you may have to upgrade everything else before restarting the nodes and the DAS. Alternatively, you can avoid these issues by disabling the data grid for every nodes. Below are the known combinations causing issues:

Known upgrade issues

Existing	Upgrade	Result
5.32.0	5.38.0	No issues
5.38.0	5.39.0	Waiting for domain1 to start .Error starting domain domain1. The server exited prematurely with exit code 1. Before it died, it produced the following output: Launching Payara Server on Felix platform ERROR: Bundle fish.payara.server.internal.core.glassfish [27] Error starting file:/home/payara/5.38/payara5/glassfish/modules/glassfish.jar (org.osgi.framework.BundleException: Unable to resolve fish.payara.server.internal.core.glassfish [27](R 27.0): missing requirement [fish.payara.server.internal.core.glassfish [27](R 27.0)] osgi.wiring.package; (&(osgi.wiring.package=org.osgi.framework)(version>=1.9.0)(!(version>=2.0.0))) Unresolved requirements: [[fish.payara.server.internal.core.glassfish [27](R 27.0)] osgi.wiring.package; (&(osgi.wiring.package=org.osgi.framework)(version>=1.9.0)(!(version>=2.0.0)))])
5.39.0	5.49.0	No issues
5.49.0	5.50.0	[#\|2025-05-29T08:28:00.518+0000\|SEVERE\|Payara 5.50.0\|com.hazelcast.security\|_ThreadID=38;_ThreadName=hz.quizzical_pike.generic-operation.thread-0;_TimeMillis=1748507280518;_LevelValue=1000;\| [192.168.122.180]:4900 [development] [5.2.2] Reason of failure for node join: Joining node's version 5.2.2 is not compatible with cluster version 4.2 (Rolling Member Upgrades are only supported for the same major version) (Rolling Member Upgrades are only supported in Hazelcast Enterprise)\|#]\|5.50.0\|5.53.0\| [#\|2025-05-29T08:45:24.146+0000\|SEVERE\|Payara 5.53.0\|com.hazelcast.security\|_ThreadID=40;_ThreadName=hz.loving_sanderson.generic-operation.thread-1;_TimeMillis=1748508324146;_LevelValue=1000;\| [192.168.122.180]:4900 [development] [5.3.1] Reason of failure for node join: Joining node's version 5.3.1 is not compatible with cluster version 5.2 (Rolling Member Upgrades are only supported in Hazelcast Enterprise)\|#]\|5.53.0\|5.58.0\| [#\|2025-05-29T10:09:27.449+0000\|SEVERE\|Payara 5.58.0\|com.hazelcast.cluster\|_ThreadID=37;_ThreadName=hz.mystifying_saha.priority-generic-operation.thread-0;_TimeMillis=1748513367449;_LevelValue=1000;\| [192.168.122.180]:4900 [development] [5.3.6] Node could not join cluster. A Configuration mismatch was detected: Incompatible partition groups! expected: HOST_AWARE, found: SPI Node is going to shutdown now!\|#]
5.50.0	5.63.0	No issues
5.63.0	5.64.0	No issues but existing user sessions get:[2025-05-29T15:06:09.342+0000] [Payara 5.65.0] [WARNING] [AS-WEB-HA-00005] [javax.enterprise.web.ha] [tid: _ThreadID=89 _ThreadName=http-thread-pool::jk-connector(1)] [timeMillis: 1748531169342] [levelValue: 900] [[ Exception occurred in getSessionjava.io.InvalidClassException: org.apache.catalina.session.StandardSession; local class incompatible: stream classdesc serialVersionUID = -5533172448459370383, local class serialVersionUID = 7816479940240670908
5.64.0	5.83.0	No issues