Configuring the Domain Data Grid underlying Hazelcast

Payara Server supports configuring Hazelcast through the Admin Console and asadmin commands. The Hazelcast configuration file can also be directly edited, but will not be covered in depth in this section.

Configuration of Hazelcast is divided into four sections. The first section applies to the whole domain, the second section applies to specific configurations and therefore to Payara Server instances that use that configuration, the third section covers how certain settings can be specified when a domain is created and the final section covers how to configure config specific Data Grid Start Port during or after the creation of an instance.

Using a Hazelcast configuration file will cause the settings set via the Admin Console and asadmin commands to be ignored. Any parameters not specified in the configuration file reverting to the Hazelcast defaults, even if they are specified in the Admin Console or domain.xml (Note - the Hazelcast defaults are not necessarily the same as the Payara Hazelcast defaults).

Since 4.1.2.172 Payara Server uses its own serializer instead of the default Hazelcast one. Setting a global serializer will not override it. This was created to avoid a bug that causes ClassNotFoundException errors.

Setting a Hazelcast Configuration File

Hazelcast configuration file can be either set using Admin Console or asadmin CLI.

Setting a Hazelcast Configuration File with the Admin Console

To set the Hazelcast configuration from a Hazelcast Configuration file:

  1. Select the domain Data Grid configuration from the page tree and click the Configuration tab to view the "Hazelcast Configuration" page:

  2. On the Hazelcast Configuration page, add the path to your Hazelcast Configuration file and click "Save". This path is relative to your domain configuration directory:

    Hazelcast Set Configuration File

Setting a Hazelcast Configuration File using asadmin

The set-hazelcast-configuration command can be used to set the configuration file:

asadmin> set-hazelcast-configuration --hazelcastconfigurationfile /path/to/file

As with the admin console, this path is relative to your domain configuration directory.

Configuring Domain Wide Hazelcast Settings with the Admin Console

Some Hazelcast configuration settings apply to the whole domain. Navigate to the _Data Grid page for the domain wide settings.

Data Grid Domain Wide Options

The following configuration options are available here:

Property Description

Restart Data Grid

Determines if the Hazelcast member embedded in Payara will be restarted to apply any changes made

DAS Bind Address

Explicity specifies the IP Address the DAS should bind the Hazelcast instance to. By default, Hazelcast will try to choose the correct bind address from the IP addresses available on the host. If you find the DAS is binding to an incorrect IP Address the correct address can be specified here.

DAS Public Address

This is the public IP Address that the DAS should use if the DAS is behind a NAT firewall and other Payara Server Instances need to connect via the public NAT address rather than the internal IP Address

Auto Increment Port

By default the cluster uses the next unoccupied port that is available starting with the start port. When auto-increment is turned off an occupied start port results in a startup failure instead.

DAS Start Port

The port the DAS uses to run Hazelcast on. If this port is busy, the port specified will be incremented until a valid port is found.

Network Interfaces

Similar to DAS Bind address by default Hazelcast will choose the correct IP Address to bind to. If you find Hazelcast is binding to the incorrect address a comma separated list of IP Addresses can be specified here.

Override Configuration File

Specifies the Hazelcast configuration file to use. The path specified is relative to the domain config directory.

If you are using a custom GlassFish server configuration for a cluster or standalone instance (e.g. cluster-config), then the Hazelcast configuration file should be placed in the directory with the same name (e.g. <domain-root>/config/cluster-config). This will ensure it is replicated to the node during startup.

Using this option to point to a valid Hazelcast configuration file will cause all other options set to be ignored. Any options unset within the Hazelcast configuration file will be set to the Hazelcast defaults.

Data Grid Group Name

Specifies the Hazelcast group name, to be optionally used to help divide clusters into logical, segregated groups (i.e. dev-group, prod-group).

Data Grid Group Password

Specifies the password used to access the Hazelcast group.

Start Port

The port the other Payara Server instances use to run Hazelcast on. If this port is busy, the port specified will be incremented until a valid port is found.

License Key

Enables Hazelcast Enterprise features.

Host Aware Partitioning

Whether or not to enable host-aware partitioning for the cluster. Host aware partitioning must be enabled on all members of the cluster (including Payara Micro instances) for this feature to work correctly. This is default in Payara Server 5.181 onwards for both Payara Server domains and Payara Micro instances.

Data Grid Discovery Mode

See Discovery Modes

Configuring Server Config Specfic Hazelcast Settings with the Admin Console

Navigate to the Data Grid page for the specific configuration.

Data Grid Domain Wide Options

The following configuration options are available here:

Property Description

Enabled

See Disabling Data Grid for detailed information

Restart Data Grid

Determines if the Hazelcast member embedded in Payara will be restarted to apply any changes made

Lite Data Grid Instance

If set to true, cluster members with this configuration will be Hazelcast "lite" nodes, meaning they will not store any cache data locally and rely on other cluster members to store data.

Config Specific Data Grid Start Port

This is only available since 5.194. Sets Data Grid Start Port to the instance. If set to 0 (default) the Domain wide start port will be used instead.

Public Address for this Data Grid Node

The Public Address to use for this Data Grid Node. Used when the node is behind a NAT

Instance Group

Instance group name for all Server Instances using the configuration. Future functionality may use this name.

Hazelcast Instance JNDI Name

The JNDI name to bind the Hazelcast instance to.

JCache Manager JNDI Name

The JNDI name to bind the JCache Cache Manager to.

JCache Caching Provider JNDI Name

The JNDI name to bind the JCache Caching Provider to.

Executor Pool Size

The threadpool size for the Hazelcast Executor service

Executor Queue Capacity

Queue Capacity of the Data Grid Executor Service. Executions are queued until a thread is available

Scheduled Executor Pool Size

The threadpool size for the Hazelcast Scheduled Executor service

Scheduled Executor Queue Capacity

Queue Capacity of the Data Grid Scheduled Executor Service. Executions are queued until a thread is available

Enter your required values, and click Save. Restarting the domain or instance/cluster is not necessary for any changes made to take effect, provided that "Dynamic" remains set to "true"

Configuring Hazelcast on Domain Creation

Since Payara Server 5.194 and 5.193.2

The following options can be used with the create-domain command to specify certain Hazelcast settings when a domain is created:

Option Type Description Default Mandatory

--hazelcastdasport

Port Number

The port to run Hazelcast on for the DAS. If this port is busy and auto-increment option is enabled, the port specified will be incremented until a valid port is found. The --hazelcastdasport option cannot be used with the --portbase option. The --hazelcastdasport option overrides the hazelcast.das.port property of the --domainproperties option.

4900

No

--hazelcaststartport

Port Number

The port the other Payara Server instances use to run Hazelcast on. If this port is busy and auto-increment option is enabled, the port specified will be incremented until a valid port is found. The --hazelcaststartport option cannot be used with the --portbase option. The --hazelcaststartport option overrides the hazelcast.start.port property of the --domainproperties option.

5900

No

--hazelcastautoincrement

Boolean

By default the Data Grid uses the next unoccupied port that is available starting with the start port. When auto-increment is turned off an occupied start port results in a startup failure instead. The --hazelcastautoincrement option overrides the hazelcast.auto.increment property of the --domainproperties option.

5900

No

The following domain properties have been added to the 'create-domain' command to make it possible to configure certain Hazelcast setting when a domain is created:

Property Description

hazelcast.das.port

This property specifies the port number of the port the DAS use to run Hazelcast on. The hazelcast.das.port property is overridden by the --hazelcastdasport option.

hazelcast.start.port

This property specifies the port number of the port the other Payara Server instances use to run Hazelcast on. The hazelcast.start.port property is overridden by the --hazelcaststartport option.

hazelcast.auto.increment

This property specifies whether or not to use the next unoccupied port that is available starting with the start port. When auto-increment is turned off an occupied start port results in a startup failure instead. The hazelcast.auto.increment property is overridden by the --hazelcastautoincrement option.
If the portbase options are used, the values for the ports of the Hazelcast DAS start port and the Hazelcast start Port are calculated as follows: Hazelcast DAS start port: portbase + 49 and Hazelcast start port: portbase + 59.

The following examples demonstrate creating a domain with all of the Hazelcast related options:

An example of create-domain command with Hazelcast specific options:

asadmin> asadmin create-domain --hazelcastdasport 7900 --hazelcaststartport 8900 --hazelcastautoincrement true testDomain

Using default port 4848 for Admin.
Using default port 8080 for HTTP Instance.
Using default port 7676 for JMS.
Using default port 3700 for IIOP.
Using default port 8181 for HTTP_SSL.
Using default port 3820 for IIOP_SSL.
Using default port 3920 for IIOP_MUTUALAUTH.
Using default port 8686 for JMX_ADMIN.
Using default port 6666 for OSGI_SHELL.
Using default port 9009 for JAVA_DEBUGGER.
Using port 7900 for Hazelcast DAS.
Using port 8900 for Hazelcast Start.
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=dev,OU=Payara,O=Payara Foundation,L=Great Malvern,ST=Worcestershire,C=UK]
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=dev-instance,OU=Payara,O=Payara Foundation,L=Great Malvern,ST=Worcestershire,C=UK]
Domain testDomain created.
Domain testDomain admin port is 4848.
Domain testDomain allows admin login as user "admin" with no password.
Command create-domain executed successfully.

An example of create-domain command with Hazelcast specific domain properties:

asadmin> create-domain --domainproperties hazelcast.das.port=7900:hazelcast.auto.increment=false:hazelcast.start.port=8900 testDomain

Using default port 8080 for HTTP Instance.
Using default port 7676 for JMS.
Using default port 3700 for IIOP.
Using default port 8181 for HTTP_SSL.
Using default port 3820 for IIOP_SSL.
Using default port 3920 for IIOP_MUTUALAUTH.
Using default port 8686 for JMX_ADMIN.
Using default port 6666 for OSGI_SHELL.
Using default port 9009 for JAVA_DEBUGGER.
Using port 7900 for Hazelcast DAS.
Using port 8900 for Hazelcast Start.
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=dev,OU=Payara,O=Payara Foundation,L=Great Malvern,ST=Worcestershire,C=UK]
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=dev-instance,OU=Payara,O=Payara Foundation,L=Great Malvern,ST=Worcestershire,C=UK]
Domain testDomain created.
Domain testDomain admin port is 4848.
Domain testDomain allows admin login as user "admin" with no password.
Command create-domain executed successfully.

Configuring Config Specific Data Grid Start Port

@Since Payara Server 5.194

It is possible to configure the port used by Hazelcast to bind the corresponding instance to the Data Grid by letting users set its value during or after the creation of an instance.

Configuring Config Specific Data Grid Start Port on Instance Creation

The following option can be used with the create-instance or the create-local-instance commands to specify config Specific Data Grid Start Port when an instance is created:

Option Type Description Default Mandatory

--dataGridStartPort

Port Number

Sets Data Grid Start Port to the instance. If set to 0 the Domain wide start port will be used instead

0

No

The following examples demonstrate creating an instance with --dataGridStartPort options:

An example of create-instance command:

asadmin> asadmin create-instance --dataGridStartPort 2900 --node localhost-domain1 instance1

Command _create-instance-filesystem executed successfully.
Port Assignments for server instance instance1:
OSGI_SHELL_TELNET_PORT=26666
JAVA_DEBUGGER_PORT=29009
JMS_PROVIDER_PORT=27676
HTTP_LISTENER_PORT=28080
IIOP_SSL_LISTENER_PORT=23820
ASADMIN_LISTENER_PORT=24848
IIOP_SSL_MUTUALAUTH_PORT=23920
JMX_SYSTEM_CONNECTOR_PORT=28686
HTTP_SSL_LISTENER_PORT=28181
IIOP_LISTENER_PORT=23700
The instance, instance1, was created on host localhost
Command create-instance executed successfully.

An example of create-local-instance command:

asadmin> asadmin create-local-instance --dataGridStartPort 2900 --node localhost-domain1 instance2

Rendezvoused with DAS on localhost:4848.
Using DAS host localhost and port 4848 from existing das.properties for node
localhost-domain1. To use a different DAS, create a new node using create-node-ssh or
create-node-config. Create the instance with the new node and correct
host and port:
asadmin --host das_host --port das_port create-local-instance --node node_name instance_name.
Port Assignments for server instance instance2:
OSGI_SHELL_TELNET_PORT=26667
JAVA_DEBUGGER_PORT=29010
JMS_PROVIDER_PORT=27677
HTTP_LISTENER_PORT=28081
IIOP_SSL_LISTENER_PORT=23821
ASADMIN_LISTENER_PORT=24849
IIOP_SSL_MUTUALAUTH_PORT=23921
JMX_SYSTEM_CONNECTOR_PORT=28687
HTTP_SSL_LISTENER_PORT=28182
IIOP_LISTENER_PORT=23701
Command create-local-instance executed successfully.

Configuring Config Specific Data Grid Start Port After Instance Creation

Config Specific Data Grid Start Port can be either set using Admin Console or asadmin CLI after an instance has been created.

Setting Config Specific Data Grid Start Port with the Admin Console

  1. Select the Configuration from the page tree, click on the relevant instance and finally click on the Data Grid.

  2. On the Configuration Specific Data Grid Configuration, set your Data Grid Start Port and click Save.

Config Specific Data Grid Start Port Option

Setting a Config Specific Data Grid Start Port using asadmin CLI

The set-hazelcast-configuration command can be used to set the Config Specific Data Grid Start Port:

asadmin> set-hazelcast-configuration --dynamic=true --configSpecificDataGridStartPort=2900 --target=instance1-config

Configuring Hazelcast using Asadmin

As noted in the enable Hazelcast section, the set-hazelcast-configuration asadmin command is used to both enable/disable Hazelcast, and to configure it. The set-hazelcast-configuration command is used to set both domain wide and config specific settings. You can pass the `--help option to the command to see usage instructions and available arguments in your terminal, both of which are listed below:

Option Type Description Default Mandatory

--enabled

Boolean

Enables Hazelcast. If no target is specified, this will enable Hazelcast on the domain configuration (server-config).

If no arguments are specified, defaults to true.

If other options are specified, asadmin will interactively prompt for --enabled.

true

No

--target

Config Name

Specifies the instance or cluster to configure.

domain

No

--dynamic

Boolean

Enable or disable dynamic stopping and starting of the embedded Hazelcast member.

false

No

-f

--hazelcastconfigurationfile

File Path

The Hazelcast configuration file to use. This path is relative to the domain config directory, <Payara Installation Directory>/glassfish/domains/<Domain Name>/config/.

Using this option to point to a valid Hazelcast configuration file will cause all other options set to be ignored. Any options unset within the Hazelcast configuration file will be set to the Hazelcast defaults.

hazelcast-config.xml

No

--startport

Port Number

The port to run Hazelcast on for Payara Server Instances. If this port is busy, the port specified will be incremented until a valid port is found.

5900

No

--daspublicaddress

IP Address

This is the public IP Address that the DAS should use if the DAS is behind a NAT firewall and other Payara Server Instances need to connect via the public NAT address rather than the internal IP Address

None

No

--dasbindaddress

IP Address

Explicity specifies the IP Address the DAS should bind the Hazelcast instance to. By default, Hazelcast will try to choose the correct bind address from the IP addresses available on the host. If you find the DAS is binding to an incorrect IP Address the correct address can be specified here.

None

No

--dasport

Port Number

The port to run Hazelcast on for the DAS. If this port is busy, the port specified will be incremented until a valid port is found.

4900

No

--publicaddress

IP Address

The Public Address to use for this Data Grid Node. Used when the node is behind a NAT

None

No

--interfaces

IP Addresses

Similar to DAS Bind address by default Hazelcast will choose the correct IP Address to bind to. If you find Hazelcast is binding to the incorrect address a comma separated list of IP Addresses can be specified here.

None

No

--clustername

Group Name

Specifies the Hazelcast group name, to be optionally used to help divide clusters into logical, segregated groups (i.e. dev-group, prod_group).

No

--clusterpassword

Password

Specifies the password used to access the Hazelcast group.

No

--hostawarepartitioning

Boolean

Whether or not to enable host-aware partitioning for the cluster. Host aware partitioning must be enabled on all members of the cluster (including Payara Micro instances) for this feature to work correctly.

false

No

--licensekey

-lk

Hazelcast License

Enables Hazelcast Enterprise features.

No

--clustermode

Cluster Mode

Can be one of domain, tcpip or multicast. See Discovery Modes for a detailed description of each mode

None

No

--multicastgroup

-g

Multicast Address

The multicast group for communications in the Hazelcast instance if multicast is selected as cluster mode.

224.2.2.3

No

--multicastport

Port Number

The multicast port for communications in the Hazelcast instance if multicast is selected as cluster mode.

54327

No

--tcpipmembers

IP Addresses

Comma separated list of IP-Address:port used to discover cluster members if tcpip is selected as the cluster mode.

None

No

--lite

Boolean

Sets the cluster to lite mode, where it will cluster but not store any cluster data.

false

No

--configSpecificDataGridStartPort

Port Number

Sets Data Grid Start Port to the instance. If set to 0 the Domain wide start port will be used instead.

0

No

--jndiname

-j

JNDI Name

Specifies the JNDI name to bind the Hazelcast instance to.

--cachemanagerjndiname

JNDI Name

Specified the JNDI name to bind the JCache Caching provider

--cachingproviderjndiname

JNDI Name

The JNDI name to bind the JCache Caching Provider to.

--executorpoolsize

Integer

The threadpool size for the Hazelcast Executor service

--executorqueuecapacity

Integer

Queue Capacity of the Data Grid Executor Service. Executions are queued until a thread is available

--scheduledexecutorpoolsize

Integer

The threadpool size for the Hazelcast Scheduled Executor service

--scheduledexecutorqueuecapacity

Integer

Queue Capacity of the Data Grid Scheduled Executor Service. Executions are queued until a thread is available

--membername

String

Member name

instance name

--membergroup

String

Instance group name for all Server Instances using the configuration. Future functionality may use this name.

config name

--help

-?

Displays the help menu.

false

No

The following example demonstrates setting all of the options on a cluster called cluster1:

asadmin> set-hazelcast-configuration --enabled=true --target=cluster1 --dynamic=true -f hazelcast-config.xml --startport=5902 -g 224.2.2.3 --multicastport=6666 --clustermode multicast --hostAwareParitioning=true -j payara/Hazelcast

Configuring Hazelcast on Domain Creation

Since Payara Server 5.194 and 5.193.2

The following options can be used with the create-domain command to specify certain Hazelcast settings when a domain is created:

Option Type Description Default Mandatory

--hazelcastdasport

Port Number

The port to run Hazelcast on for the DAS. If this port is busy and auto-increment option is enabled, the port specified will be incremented until a valid port is found. The --hazelcastdasport option cannot be used with the --portbase option. The --hazelcastdasport option overrides the hazelcast.das.port property of the --domainproperties option.

4900

No

--hazelcaststartport

Port Number

The port the other Payara Server instances use to run Hazelcast on. If this port is busy and auto-increment option is enabled, the port specified will be incremented until a valid port is found. The --hazelcaststartport option cannot be used with the --portbase option. The --hazelcaststartport option overrides the hazelcast.start.port property of the --domainproperties option.

5900

No

--hazelcastautoincrement

Boolean

By default the Data Grid uses the next unoccupied port that is available starting with the start port. When auto-increment is turned off an occupied start port results in a startup failure instead. The --hazelcastautoincrement option overrides the hazelcast.auto.increment property of the --domainproperties option.

5900

No

The following domain properties have been added to the 'create-domain' command to make it possible to configure certain Hazelcast setting when a domain is created:

Property Description

hazelcast.das.port

This property specifies the port number of the port the DAS use to run Hazelcast on. The hazelcast.das.port property is overridden by the --hazelcastdasport option.

hazelcast.start.port

This property specifies the port number of the port the other Payara Server instances use to run Hazelcast on. The hazelcast.start.port property is overridden by the --hazelcaststartport option.

hazelcast.auto.increment

This property specifies whether or not to use the next unoccupied port that is available starting with the start port. When auto-increment is turned off an occupied start port results in a startup failure instead. The hazelcast.auto.increment property is overridden by the --hazelcastautoincrement option.
If the portbase options are used, the values for the ports of the Hazelcast DAS start port and the Hazelcast start Port are calculated as follows: Hazelcast DAS start port: portbase + 49 and Hazelcast start port: portbase + 59.

The following examples demonstrate creating a domain with all of the Hazelcast related options:

An example of create-domain command with Hazelcast specific options:

asadmin> asadmin create-domain --hazelcastdasport 7900 --hazelcaststartport 8900 --hazelcastautoincrement true testDomain

Using default port 4848 for Admin.
Using default port 8080 for HTTP Instance.
Using default port 7676 for JMS.
Using default port 3700 for IIOP.
Using default port 8181 for HTTP_SSL.
Using default port 3820 for IIOP_SSL.
Using default port 3920 for IIOP_MUTUALAUTH.
Using default port 8686 for JMX_ADMIN.
Using default port 6666 for OSGI_SHELL.
Using default port 9009 for JAVA_DEBUGGER.
Using port 7900 for Hazelcast DAS.
Using port 8900 for Hazelcast Start.
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=dev,OU=Payara,O=Payara Foundation,L=Great Malvern,ST=Worcestershire,C=UK]
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=dev-instance,OU=Payara,O=Payara Foundation,L=Great Malvern,ST=Worcestershire,C=UK]
Domain testDomain created.
Domain testDomain admin port is 4848.
Domain testDomain allows admin login as user "admin" with no password.
Command create-domain executed successfully.

An example of create-domain command with Hazelcast specific domain properties:

asadmin> create-domain --domainproperties hazelcast.das.port=7900:hazelcast.auto.increment=false:hazelcast.start.port=8900 testDomain

Using default port 8080 for HTTP Instance.
Using default port 7676 for JMS.
Using default port 3700 for IIOP.
Using default port 8181 for HTTP_SSL.
Using default port 3820 for IIOP_SSL.
Using default port 3920 for IIOP_MUTUALAUTH.
Using default port 8686 for JMX_ADMIN.
Using default port 6666 for OSGI_SHELL.
Using default port 9009 for JAVA_DEBUGGER.
Using port 7900 for Hazelcast DAS.
Using port 8900 for Hazelcast Start.
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=dev,OU=Payara,O=Payara Foundation,L=Great Malvern,ST=Worcestershire,C=UK]
Distinguished Name of the self-signed X.509 Server Certificate is:
[CN=dev-instance,OU=Payara,O=Payara Foundation,L=Great Malvern,ST=Worcestershire,C=UK]
Domain testDomain created.
Domain testDomain admin port is 4848.
Domain testDomain allows admin login as user "admin" with no password.
Command create-domain executed successfully.