Configuring Java Message Service High Availability

This chapter describes how to configure the high availability features of the Java Message Service (JMS). It covers how to configure Message Queue broker clusters and how to use them to provide connection failover and load balancing.

It is important to be aware of the distinction between a message broker cluster (a group of broker instances) and a Payara Server cluster (a traditional group of server instances) to fully understand the concepts and instructions described in this chapter.

Documentation for the Open Message Queue software can be found in the following online resources:

Using Message Queue Broker Clusters With Payara Server

This section describes how the JMS service uses Message Queue broker clusters to support high-availability JMS messaging in Payara Server clusters and deployment groups. It describes the different cluster and broker types that are supported and how to configure them.

About Message Queue Broker Clusters

The following discussion provides a brief overview of Message Queue broker clusters.

Message Queue supports two clustering models both of which provide a scalable message service, but with each providing a different level of message service availability:

Conventional broker clusters

A conventional broker cluster provides for service availability. When a broker fails, clients connected to the failed broker reconnect to another broker in the cluster.

However, messages and state information stored in the failed broker cannot be recovered until the failed broker is brought back online.

The broker failure can therefore result in a significant delay and in JMS message order semantics not being preserved.

Message Queue supports two types of conventional cluster, based on where the cluster configuration change record is stored:

Conventional cluster with master broker: In a conventional cluster with a master broker, one of the brokers, designated as the master broker, stores and maintains the cluster configuration change record. The other brokers in the cluster must communicate with the master broker to keep abreast of changes to the cluster configuration.

This is the simplest broker cluster to configure, and is the type of broker cluster that Payara Server uses by default.
Conventional cluster of peer brokers: In a conventional cluster of peer brokers, the cluster configuration change record is stored in a JDBC data store accessible to all the brokers.

Thus, brokers can access cluster configuration information whether any other brokers in the cluster are running or not.

Enhanced broker clusters

An enhanced broker cluster provides for data availability in addition to service availability. When a broker fails, another broker takes over the pending work of the failed broker.

The failover broker has access to the failed broker’s messages and state information. Clients connected to the failed broker reconnect to the failover broker.

In an enhanced cluster, as compared to a conventional cluster, messages owned by the failed broker are delivered by the failover broker as soon as it takes over, and JMS message order semantics are preserved.

By its very nature, an enhanced broker cluster is always cluster of peer brokers.

Despite the message service availability offered by both conventional and enhanced broker clusters, they do not provide a guarantee against failure and the possibility that certain failures, for example in the middle of a transaction, could require that some operations be repeated.
It is the responsibility of the messaging application (both producers and consumers) to respond to JMS exceptions appropriately.

Configuring Payara Server Clusters to Use Message Queue Broker Clusters

When a Payara Server cluster is created, the JMS service automatically configures a Message Queue conventional broker cluster with master broker for the cluster, provided that the JMS host type in the Payara Server cluster’s configuration is either Embedded or Local.

The JMS service configures one Message Queue broker for each instance in the Payara Server cluster, and designates as master broker the broker associated with the first instance created in the cluster.

In the case of Embedded JMS hosts, each broker inherently runs on the same host as the instance with which it is associated because it runs in the same JVM as the instance.
In the case of Local JMS hosts, the JMS service configures each broker to run on the same host as the instance with which it is associated.

The JMS service manages the lifecycle of Embedded and Local JMS hosts, and this management extends to the management of Message Queue broker clusters as Embedded and Local JMS hosts. For a Payara Server cluster whose configuration specifies Embedded or Local JMS host type, the JMS service:

Creates and manages ONE Message Queue broker for each instance in the Payara Server cluster, using this broker instance as the primary JMS host for the instance, keeping a one-to-one relationship between both.
Maintains the JMS host list for each instance in the Payara cluster such that its primary JMS host appears first in its JMS host list.

The JMS service supports the following types of Message Queue broker clusters with Payara Server clusters, based on the JMS host type:

Embedded

Conventional broker cluster with master broker (default)
Conventional broker cluster of peer brokers

Local

Conventional broker cluster with master broker (default)
Conventional broker cluster of peer brokers
Enhanced broker cluster

Remote

Conventional broker cluster with master broker.

Brokers can differ in number from Payara Server instances and can be located on other hosts
Conventional broker cluster of peer brokers.

Brokers can differ in number from Payara Server instances and can be located on other hosts
Enhanced broker cluster.

Brokers can differ in number from Payara Server instances and can be located on other hosts.

It is recommended that in a robust production environment configuration, the JMS service must be configured in either Local or Remote mode.
The Embedded mode is not recommended for the usual reliability and scalability that a production system may require.

The following topics provide instructions for configuring broker clusters in all these contexts.

Deployment Groups Considerations

The instructions provided will also apply to deployment groups as well, however, it is mandatory that all Payara Server instances reference the same named configuration object. If multiple instances in the same deployment group reference multiple, separate configuration objects, high availability for the JMS service won’t be guaranteed.

When running the commands listed in the respective instructions, instead of targeting a server cluster, you can target a deployment group instead.

To Configure a Payara Cluster to Use an Embedded or Local Conventional Broker Cluster With Master Broker

Use the configure-jms-cluster subcommand in remote asadmin mode to configure a conventional broker cluster with master broker to service a Payara Server cluster that uses either Embedded or Local JMS hosts.

Note that the Embedded mode brokers configuration is the default for Payara Server clusters.

Before You Begin: Perform the following steps after you have created the Payara Server cluster, but before you have added instances to the cluster or started the cluster.

Before using this procedure to reconfigure an existing cluster, you must follow the special procedures to migrate to another type of broker cluster, as described in To Migrate Between Types of Embedded or Local Conventional Broker Clusters.
Failing to perform these special procedures could lead to data loss or corruption and even render your setup unusable, depending on the JMS operations performed on the existing cluster.

Ensure that the server is running. Remote asadmin subcommands require a running server.
Configure the Payara Server cluster to use a Message Queue conventional broker cluster with master broker by using the configure-jms-cluster subcommand:
```
asadmin configure-jms-cluster --clustertype=conventional --configstoretype=masterbroker payara-cluster-name
```

See Also

You can also view the full syntax and options of the subcommand by typing asadmin help configure-jms-cluster at the command line.

To Configure a Payara Server Cluster to Use an Embedded or Local Conventional Broker Cluster of Peer Brokers

Use the configure-jms-cluster subcommand in remote asadmin mode to configure a conventional broker cluster of peer brokers to service a Payara Server cluster that uses Embedded or Local JMS hosts.

Before You Begin: Perform the following steps after you have created the Payara Server cluster, but before you have added instances to the cluster or started it.

Ensure that the server is running. Remote asadmin subcommands require a running server.
Create a password file with the entry AS_ADMIN_JMSDBPASSWORD specifying the password of the database user.

For information about password file entries, see the asadmin help page.
Place a copy of, or a link to, the database’s JDBC driver .jar file in the appropriate directory, depending on the JMS host type, on each host where a Payara Server cluster instance is to run:
- Embedded: as-install-parent/payara/lib/install/applications/jmsra
- Local: as-install-parent/mq/lib/ext

Configure the Payara Server cluster to use a Message Queue conventional broker cluster with master broker by using the configure-jms-cluster subcommand:

asadmin --passwordfile password-file configure-jms-cluster --clustertype=conventional --configstoretype=shareddb --dbvendor database-vendor-name --dbuser database-user-name --dburl database-url --property list-of-database-specific-properties <server-cluster-name>

See Also

You can also view the full syntax and options of the subcommand by typing asadmin help configure-jms-cluster at the command line.

To Change the Master Broker in an Embedded or Local Broker Cluster

Use the change-master-broker subcommand in remote asadmin mode to change the master broker to a different broker in a conventional broker cluster with master broker serving a Payara Server cluster that uses Embedded or Local JMS hosts.

Follow this procedure, for example, before you remove from a Payara Server cluster the instance associated with the current master broker.

Before You Begin: Although not an absolute requirement, you should make sure all Payara Server instances and Message Queue brokers in the cluster are running before using the change-master-broker command in order to avoid later internal configuration synchronization of any unavailable instance or broker.

Ensure that the server is running.

Remote asadmin subcommands require a running server.
Change the master broker by using the change-master-broker subcommand:
```
asadmin change-master-broker payara-clustered-instance-name
```

See Also

You can also view the full syntax and options of the subcommand by typing asadmin help change-master-broker at the command line.

To Migrate Between Types of Embedded or Local Conventional Broker Clusters

If the need arises to convert from a conventional broker cluster with master broker to a conventional broker cluster of peer brokers, or the reverse, follow the instructions in the Managing Conventional Clusters section of the OpenMQ Administration Guide.

To Configure a Payara Server Cluster to Use a Local Enhanced Broker Cluster

Use the configure-jms-cluster subcommand in remote asadmin mode to configure an enhanced broker cluster to service a Payara Server cluster that uses Local JMS hosts.

Before You Begin: Perform the following steps after you have created the Payara Server cluster, but before you have added instances to the cluster or started it.

Before using this procedure to reconfigure an existing cluster, you must follow the special procedures to migrate from a conventional broker cluster to an enhanced broker cluster, as described in the Converting a Conventional Cluster to an Enhanced Cluster section in the OpenMQ Administration Guide.
Failing to perform these special procedures could lead to data loss or corruption and even render your setup unusable, depending on the JMS operations performed on the existing cluster.

Ensure that the server is running. Remote asadmin subcommands require a running server.
Create a password file with the entry AS_ADMIN_JMSDBPASSWORD specifying the password of the database user.

For information about password file entries, see the asadmin help page.
Place a copy of, or a link to, the database’s JDBC driver .jar file in the as-install-parent/mq/lib/ext directory on each host where a Payara Server cluster instance is running.

Configure the Payara Server cluster to use a Message Queue enhanced broker cluster by using the configure-jms-cluster subcommand:

asadmin --passwordfile password-file configure-jms-cluster --clustertype=enhanced --configstoretype=shareddb --messagestoretype=jdbc --dbvendor database-vendor-name --dbuser database-user-name --dburl database-url --property list-of-database-specific-properties payara-cluster-name

See Also

You can also view the full syntax and options of the subcommand by typing asadmin help configure-jms-cluster at the command line.

To Configure a Payara Server Cluster to Use a Remote Broker Cluster

Before You Begin

Perform the following steps after you have:

Used Message Queue to create a broker cluster.
Created the Payara Server cluster, but not yet created instances for the cluster.
1. Ensure that the server is running. The remote subcommands used in this procedure require a running server.
2. Delete the default_JMS_host JMS host by using the delete-jms-host subcommand:
  asadmin delete-jms-host --target payara-cluster-name default_JMS_host
3. Create a JMS host for each broker in the broker cluster by using the create-jms-host subcommand.
  
  For each broker, use an asadmin create-jms-host command instruction like this:
  asadmin create-jms-host --target payara-cluster-name --mqhost broker-host --mqport broker-port --mquser mq-user --mqpassword mq-user-password <jms-host-name-for-broker>
4. Start the brokers in the cluster by using the Message Queue imqbrokerd command, as described in the Managing Broker Clusters section the in OpenMQ Administration Guide.
5. Create instances in the Payara Server cluster, as described in To Create an Instance Centrally and To Create an Instance Locally.

Connection Failover

The use of Message Queue broker clusters provides JMS connection failover, including several options that control how connection failures are handled.

Use the Administration Console’s Java Message Service page to configure these options. To display this page, click the configuration for the Payara server cluster or instance’s configuration in the navigation pane, and then click the Java Message Service link on the Configuration page.

The way in which connection failover operates depends on whether the broker cluster is configured to be conventional or enhanced:

In a conventional cluster, when a broker fails, clients may reconnect to any other broker in the cluster.

The Reconnect field specifies whether reconnection should take place, and the Address List Behavior and Address List Iterations fields specify how the client chooses what broker to fail over to.
In an enhanced cluster, when a broker fails, another broker automatically takes over its messages and clients.

Clients automatically fail over to the appropriate broker. The Reconnect, Address List Behavior and Address List Iterations fields are ignored.

For more information on connection failover, including how failover on conventional clusters differs from failover on enhanced clusters, see the Automatic Reconnection in the OpenMQ Administration Guide.

Reconnect

Applies only to conventional clusters. Enables reconnection and connection failover.

When disabled, the Java Message Service does not attempt to reconnect if a connection fails.

Reconnect Interval

Specifies the number of seconds between reconnection attempts.

If it is too short, this time interval does not give a broker time to recover.

If it is too long, the wait time might represent an unacceptable delay.

The default value is 5 seconds.

Reconnect Attempts

Specifies the number of attempts to connect (or reconnect) to a particular JMS host before trying another host in the JMS host list. The host list is also known as the Address List.

Hosts are chosen from the address list either in order or randomly, depending on the setting of Address List Behavior.

Address List Behavior

For conventional clusters, this field specifies how the Java Message Service selects which JMS host in the JMS hosts list to initially connect to, and if the broker fails, how the Java Message Service selects which JMS host in the JMS hosts list to fail over to.

For enhanced clusters, this field specifies how the Java Message Service selects which JMS host in the JMS hosts list to initially connect to.

When performing initial connection or, for conventional clusters only, when performing failover, then if this attribute is set to Priority, the Java Message Service tries to connect to the first JMS host specified in the JMS hosts list and uses another one only if the first one is not available. If this attribute is set to Random, the Java Message Service selects the JMS host randomly from the JMS hosts list. If that host is not available, another one is chosen randomly.

The default value for Embedded and Local JMS host types is Priority, and the default for the Remote JMS host type is Random.

For Embedded and Local JMS host types, the Java Message Service ensures that the Message Queue broker servicing a clustered instance appears first in that instance’s JMS host list.

Thus, having Priority as the default Address List Behavior ensures that an application deployed to a clustered instance will always try to create its initial connection to that instance’s co-located broker.

If there are many clients attempting a connection using the same connection factory, use the Random setting to prevent them from all attempting to create their initial connection to the same JMS host.

Address List Iterations

For conventional clusters, this field specifies the number of times the Java Message Service iterates through the JMS hosts list in an effort to establish its initial connection.

If the broker fails, this field specifies the number of times the Java Message Service iterates through the JMS hosts list in an effort to fail over to another broker.

For enhanced clusters, this field specifies the number of times the Java Message Service iterates through the JMS hosts list in an effort to establish its initial connection. If the broker fails, this field is not used when performing reconnection.

You can override these settings using JMS connection factory settings. For details, see "Administering JMS Connection Factories and Destinations" in the Payara Server General Administration section.

Load-Balanced Delivery to MDBs

When a message-driven bean (MDB) application is deployed to a Payara Server cluster, incoming messages are delivered randomly to MDBs without regard to the server cluster instances in which they are running.

If the MDB is configured to receive messages from a durable or non-durable subscription on a topic, then only one MDB instance across the whole Payara Server cluster will receive each message.

For more information about these features, see the About Shared Topic Subscriptions for Clustered Containers section in the OpenMQ Administration Guide.