Monitoring Console

To switch page use the drop-down menu at the top centre of the page.

Since version 5.20.0, each page switch also updates the URL making the URL bookmarkable. URLs can be copied and shared with others to share particular pages with them.

Updating URLs after switching pages also means that once the page is changed using the dropdown navigation, the browser’s page history based navigation can be used to go back to already visited pages.

Each page uses a column layout of 1-4 columns. The number of columns for a page can be changed using top right Layout menu.

Changing the column layout of a page does not change the preferred position of the widgets. If a widget is set to column 3 but a 2-column layout is used all widgets of the 3rd or 4th column get pushed to the 2nd column instead. This allows switching layouts without redoing the configuration entirely.

By default the graphs on a page update every 2 seconds. To control the interval use top right Data Refresh menu or open the Settings and change the interval in the Global settings.

The interval can be paused and continued without changing the interval duration by using the paused checkbox next to the interval setting.

To add a new page use the top right Page menu and select New…​.

This creates a new Unnamed page which can be renamed in the Page settings.

After naming the page add widgets.

To delete a page open top right Page menu, select Delete and confirm the browser confirmation dialogue.

When the configuration of a preconfigured page or any of the widgets on it was changed by the user the page can be reset by selecting Reset from the top right Page menu.

To reset all pages and remove all user created pages clear the browser’s local storage for the monitoring console page.

Monitoring Console uses the browser’s local storage to store the page configuration (which includes the widgets placed on a page). This means by default changes to page configurations do not affect other users. To make pages available to others and to start from a common base configuration pages are explicitly synchronised with a persistent server page configuration.

The chosen role controls the behaviour and available options for the page synchronisation. It is the Administrator's role that is meant to manage the persistent server page configuration. The Administrator pushes local changes to the server to update the base configuration. Users and Guests pull the server configuration to update their local configuration with the base. The control remains with each user.

The Page Synchronisation dialogue shown when entering the console UI allows the user to pick the pages that should be updated (pulled).

Based on the role the pages to synchronise are pre-selected.

The newest version is marked by a green background for easier orientation.

This dialogue can also be opened from the Global settings by clicking the Update Local Pages button shown Page Sync property.

Administrators also find a button to Update Remote Pages with any changes applied to local changes. This is only needed for pages that are not using _auto_matic updates.

To share a page that so far is not part of the persistent remote configuration open the Page settings and look for the Sync property:

Users and Administrators can both Push a page to update the persistent remote configuration and Pull the remote configuration to update the local page. Administrators can also enable or disable the automatic page synchronisation. When enabled any local change done by an Administrator is automatically pushed to the remote configuration and can be pulled by other users.

The monitoring console can be used in one of three roles:

When no role is set for the session the user is asked to choose a role when opening the monitoring console.

The selection can be changed at any time in the Global settings for Role.

Widgets can be added to any page. Open the side panel by clicking on Settings top right menu. Use the dropdown in the Add Widgets property of the Page settings to select the category of the metric to add. This populates the second dropdown with all metrics for that category. Select a metric from the second dropdown to pre-fill the widget series name field below and press the Add button.

When added, widgets are automatically placed in a column so that they are evenly distributed on the page. Use the side panel widget settings to change column, span or item position of the widget or use the quick menu for the widget by clicking the cog symbol in the top right corner of the widget and select the action you want to apply to the widget.

To configure a widget select it by clicking on its title.

This opens the side panel and shows the detail configuration of the widget. The side panel can be closed by either deselecting the widget, by clicking on its title again, or by clicking the Settings menu or its Hide item.

The Widget settings are concerned with the widgets position within the page.

The Data settings are concerned with what and how the data is shown in the graph of a widget.

Decorations are visual helpers that can be added to a graph of a widget in order to make it easier to for the user to quickly understand the data displayed. In particular these are reference lines.

Besides adding a reference line to the graph the critical and alarming thresholds affect the status evaluation.

Use the colour input to individually override the widget’s colour for waterline, alarming line or critical line in graphs. Press the button right next to it which shows the default colour as configured in the Colors settings to reset the individual colour to use the default setting instead. A colour set to the default changes with the default.

The current value of a metric can automatically be categorised as a certain status. For example based on the threshold values of decorations or by encountered errors or missing data. The Status settings allows to set a custom message for a particular status.

The messages support simple markup using *bold* and _italic_.

The values of metric series can be tracked by the server to report values that out of a healthy range. This is represented by states:

Alerts can occur when a watch is configured for a metric series. A watch describes the conditions and thresholds used to determine a state from the recent values of a metric series. It is either Red (unhealthy), Amber (degraded), Green (healthy) or White (undetermined). A Red or Amber state causes an alert that is shown on top of the graph.

When an alert occurred for a metric shown in a graph the background of the line causing the alert is coloured in Red or Amber default colour until the alert is stopped.

Alerts are considered ongoing as long as the state is Red or Amber. If it is evaluated to Green or White the alert stops. Stopped alerts are no longer shown on top of line graphs. Another way to remove an alert from the line graph is to acknowledge the alert by checking its checkbox. To see even stopped or acknowledged alerts add a widget of type Alert Table.

By default an alert table shows all alerts including stopped and acknowledged alerts. Use the widget Alerts settings to exclude alerts with certain properties.

Alert table shows the overall state of each instance in the table’s legend.

Alerts settings can equally be used for line graphs to create a more restrictive setting.

Any widget can be removed from any page. Click the cog symbol in the top right of the widget, select Remove and confirm the browser dialogue.

Remember that a page originating from a preset can be reset to recover a widget that was removed accidentally.

A series is a name or key for a particular metric shown in a widget. Each metric has a unique key or series name. Usually these names use tags to categorise metrics into groups. For example:

The patterns shown above exemplify conventions used throughout Payara monitoring console. The first grouping level usually is the ns (short for namespace), the second the @ (short for group). In the example the metrics belong to the jvm namespace and the groupName group within that namespace.

When metrics are selected using the dropdowns these details are filled in for the user by selecting items from the dropdowns. Alternatively, a user may choose to enter the series key manually which allows using a wild-card star instead of either a tag value (like jvm) or the MetricName. For example:

This will then match all existing series that match the pattern. This feature can be used to automatically show all metrics of a certain pattern in a single widget as it is done by the request tracing page.

A configuration contains all pages. All changes made to a page or widget setup are contained in this configuration. To download this configuration as JSON file select Export from the Settings top right menu.

To import a previously exported JSON file select Import in the top right Settings menu and select the JSON file to import. This feature can be used to version and distribute configurations for similar installations.

Monitoring console is built with the intention of being used to display metrics on status monitors. To allow viewing a variety of pages and their metrics on a single screen the pages can be automatically rotated showing each page for a configured interval duration.

To enable page rotation open the Settings and check the enabled checkbox in the Global Settings Page Rotation property. This starts the rotation immediately.

The duration can be adjusted changing the text field of the Page Rotation to another time value. Use units to express the duration, as for example: 1min, 30sec. Changes take effect immediately.

The rotation will show each page for the configured duration. Pages can be excluded from rotation by switching to the page using the page menu. Open the Settings and uncheck the Include in Rotation checkbox in the Page settings.

To create custom watches go to the Watches page using the top page menu or the Global settings.

The above example shows a relatively simple watch named HTTP server errors. The name of the watch can be anything as long as it is unique. The watch should have both a Unhealthy and a Degraded level.

The field after the If holds the name of the metric series the watch is linked to, in this case the ns:http ServerCount5xx metric, as shown on the HTTP page. The in clause specifies the unit type of the given metric, here it is the Count of 5xx responses. Following is the comparison operator is selected, here >. This is followed by the threshold value, here 3 or 1, and the scope of the comparison, here in sample. As both do not use a stop condition the checkbox after until is not checked.

The below table describes possible choices for each step of a watch condition.

The below table describes the semantics of the different types of comparison scopes.

After composing the watch conditions press Save or Update to install the watch. If a watch of the same name already exists the existing watch is replaced. Watches that originate from server modules cannot be replaced. To change them disable the existing watch and click to edit it, this will create a duplicate with a different name that can be adjusted freely.

To edit a previously created watch click on its name in the list of watches or chose Edit from the cogs menu on the right.

Watches that originate from server modules cannot be edited. When clicking their names or when choosing Duplicate from their cogs menu a duplicate with a different name is created. The suggested name Copy of {original name} can be changed as long as it is different to existing watch names.

When changing the name of an edited watch to an existing name this overrides that watch. In other words watches are always identified by their name.

Go to the Watches page using the top page menu or the Global settings.

The status of each watch is reflected by the checkbox left of the watches name, checked for enabled, unchecked for disabled. Disabled or stopped watches names are also greyed out. To disable an enabled watch un-check the checkbox or use the Disable item from the cogs menu on the right of the watch widget. To enable a disabled watch check the checkbox or use the Enable item from the cogs menu. The change has immediate effect.

The Core page gives an overview of some of the important instance metrics. Each instance in the grid is shown separately.

The HTTP page gives a rough overview of the state of the HTTP services of each instance.

The JVM pages gives a general overview of the state of the JVM running Payara Server Enterprise.

A page that shows key metrics on the monitoring system itself that feeds the monitoring console with data. This is mostly useful to identify problems with the monitoring system or to cross check that unexpected metric values aren’t caused by general problems with monitoring.

A page that gives an overview of the state of the server health checks. The individual checks need to be enabled in the server HealthCheck configuration.

The Threads page gives detailed information on thread health of the cluster and lists information on stuck or hogging threads. Total number of live and daemon threads are given for reference.

The Request Tracing page overview page shows traces that exceeded the set threshold.

The Trace Duration Range widget shows all traces for the same method as a single bar. The length and position of the bar gives the range of total trace duration for all the occurrences that exceeded the threshold. The bar starts at the minimum duration observed and ends at the maximum duration observed in the last minute. The legend shows the average duration and the method traced (innermost trace span operation)

When all time minimum and/or maximum are added in the Widget Data settings the bar starts and/or ends at these "all time" values instead of the minimum and/or maximum within the last minute.

The Trace Duration Above Threshold widget shows a graph where each point marks the duration of a "slow" request at the moment the requests occurred. Possible alerts are shown on top of the graph.

The Trace Data view adds tracing details for the "slow" requests that can be sorted by wall-time or value using the widget menu (gear icon in top right corner of the widget).

By clicking on one of the bars in the overview’s Trace Duration Range widget the details of all kept traces for this method can be viewed. This data corresponds to the traces stored in the trace-store. For each trace, each of the spans corresponds to a bar in this graph.

By default the traces are sorted longest (slowest) to shortest (fastest) total duration. On the time axis the spans are shown relative to the start of the trace.

When the side panel is opened the details of each trace span can be viewed by clicking on the span bar.

The legend gives average durations for each of the spans occurring for the traced method.

The spans can be sorted by wall time by selecting Sort by Wall Time or by duration by selecting Sort By Duration in the Sorting menu in the top right of the widget.

To get back to the overview close the details by clicking the x-icon on the right side of the Sorting menu.

The SQL page gives insight into slow SQL queries for all connection pools configured in Payara Server Enterprise.

The main use of the Alerts page is to check for any server alerts. The page is split into two lists: the upper list containing all ongoing alerts, and the lower list containing past alerts. As always for alert lists both lists are sorted starting with the most recent severe alert progressing to the oldest least severe alert. Annotations associated with alerts are hidden in this view to make room for more alert entries.

Past alerts can be managed by acknowledging the alerts which makes them disappear from the list.