Monitoring Console

To switch page use the drop-down menu at the top centre of the page.

Updating URLs after switching pages also means that once the page is changed using the dropdown navigation, the browser’s page history based navigation can be used to go back to already visited pages.

Each page uses a column layout of 1-4 columns. The number of columns for a page can be changed using top right Layout menu.

Changing the column layout of a page does not change the preferred position of the widgets. If a widget is set to column 3 but a 2-column layout is used all widgets of the 3rd or 4th column get pushed to the 2nd column instead. This allows switching layouts without redoing the configuration entirely.

By default, the graphs on a page update every 2 seconds. To control the interval use top right Data Refresh menu or open the Settings and change the interval in the Global settings.

The interval can be paused and continued without changing the interval duration by using the paused checkbox next to the interval setting.

To add a new page use the top right Page menu and select New…​.

This creates a new Unnamed page which can be renamed in the Page settings.

After naming the page add widgets.

To delete a page open-top right Page menu, select Delete and confirm the browser confirmation dialogue.

When the configuration of a preconfigured page or any of the widgets on it was changed by the user the page can be reset by selecting Reset from the top right Page menu.

To reset all pages and remove all user created pages clear the browser’s local storage for the monitoring console page.

The Monitoring Console uses the browser’s local storage to store the page configuration (which includes the widgets placed on a page). This means by default changes to page configurations do not affect other users.

To make pages available to others and to start from a common base configuration pages are explicitly synchronised with a persistent server page configuration.

The monitoring console can be used in one of three roles:

The chosen role controls the behaviour and available options for the page synchronisation:

The control remains with each user.

The Page Synchronisation dialogue shown when entering the console UI allows the user to pick the pages that should be updated (pulled).

Based on the role the pages to synchronise are pre-selected.

The newest version is marked by a green background for easier orientation.

This dialogue can also be opened from the Global settings by clicking the Update Local Pages button shown Page Sync property.

Administrators also find a button to Update Remote Pages with any changes applied to local changes.

To share a page that so far is not part of the persistent remote configuration open the Page settings and look for the Sync property:

Users and Administrators can both Push a page to update the persistent remote configuration and Pull the remote configuration to update the local page. Administrators can also enable or disable the automatic page synchronisation.

When no role is set for the session the user is asked to choose a role when opening the monitoring console.

The selection can be changed at any time in the Global settings for Role.

The individual page settings are explained in more detail in the context of the operations they are involved in. This table gives an additional overview.

Widgets can be added to any page. Open the side panel by clicking on Settings top right menu.

Make sure the page is of Type Manual. Pages of type Query can be changed to Manual as well which keeps them as they are for further manual customisation.

Use the Select metric(s) button of the Add Widgets property of the Page settings to open the metric selection dialog.

In the selection dialog use the filters to find the metric you are looking for.

When the metric is found in the list of matches check it to add it to the selection. After selecting one or more metrics to show in the widget confirm the selection by pressing Ok button of the dialog.

Filters that cannot be used in combination with chosen options of other filters are not shown and have no effect. They reappear as soon as they can be used.

When added, widgets are automatically placed in a column so that they are evenly distributed on the page. Use the side panel widget settings to change column, span or item position of the widget or use the quick menu for the widget by clicking the cog symbol in the top right corner of the widget and select the action you want to apply to the widget.

The second form of populating a page is to not manually add widgets to it but to define a series query. Matching metrics will be used to automatically populate the page with widgets.

To use this option change the page Type to Query in the Page settings.

At a minimum the Query Series has to be defined. This should be a series using a pattern containing wild-cards (see Series with Wildcards for more details).

The matching series are ordered and placed on the page in intervals. The interval is controlled by the Query Interval property. Usually the interval is between 1 minute and 1 hour. If no interval is given the page will first update after 365 days.

The Update button can be used to force a page update before the next interval starts and ends. Changes to the Query Series, Query Interval or Max Size do not have immediate effect and can be applied immediately using the Update button.

Depending on the Query Series and its potential to match vast numbers of metrics it can be a good idea to limit the maximum number of widgets automatically placed on the page by the population from query using the Max Size property.

To configure a widget select it by clicking on its title.

This opens the side panel and shows the detail configuration of the widget. The side panel can be closed by either deselecting the widget, by clicking on its title again, or by clicking the Settings menu or its Hide item.

The Widget settings are concerned with the widgets position within the page.

The Data settings are concerned with what and how the data is shown in the graph of a widget.

Decorations are visual helpers that can be added to a graph of a widget in order to make it easier to for the user to quickly understand the data displayed.

In particular these are reference lines.

Besides adding a reference line to the graph the critical and alarming thresholds affect the status evaluation.

Use the colour input to individually override the widget’s colour for waterline, alarming line or critical line in graphs.

Press the button right next to it which shows the default colour as configured in the Colors settings to reset the individual colour to use the default setting instead. A colour set to the default changes with the default.

The current value of a metric can automatically be categorised as a certain status. For example based on the threshold values of decorations or by encountered errors or missing data. The Status settings allows to set a custom message for a particular status.

The messages support simple markup using *bold* and _italic_.

The values of metric series can be tracked by the server to report values that out of a healthy range.

This is represented by these states:

Alerts can occur when a watch is configured for a metric series.

A watch describes the conditions and thresholds used to determine a state from the recent values of a metric series. It is either Red (unhealthy), Amber (degraded), Green (healthy) or White (undetermined).

A Red or Amber state causes an alert that is shown on top of the graph.

When an alert occurred for a metric shown in a graph the background of the line causing the alert is coloured in Red or Amber default colour until the alert is stopped.

Alerts are considered ongoing as long as the state is Red or Amber. If it is evaluated to Green or White the alert stops. Stopped alerts are no longer shown on top of line graphs. Another way to remove an alert from the line graph is to acknowledge the alert by checking its checkbox.

To see stopped or acknowledged alerts add a widget of type Alert Table.

By default, an alert table shows all alerts including stopped and acknowledged alerts. Use the widget Alerts settings to exclude alerts with certain properties.

Alert table shows the overall state of each instance in the table’s legend.

Alerts settings can equally be used for line graphs to create a more restrictive setting.

Any widget can be removed from any page. Click the cog symbol in the top right of the widget, select Remove and confirm the browser dialogue.

Remember that a page originating from a preset can be reset to recover a widget that was removed accidentally.

A series is a name or key for a particular metric shown in a widget. Each metric has a unique key or series name. Usually these names use tags to categorise metrics into groups. For example:

The patterns shown above exemplify conventions used throughout Payara monitoring console. The first grouping level usually is the ns (short for namespace), the second the @ (short for group).

In the example the metrics belong to the jvm namespace and the groupName group within that namespace.

When metrics are selected using the dropdowns these details are filled in for the user by selecting items from the dropdowns. Alternatively, a user may choose to enter the series key manually which allows using a wild-card star instead of either a tag value (like jvm) or the MetricName. For example:

This will then match all existing series that match the pattern. This feature can be used to automatically show all metrics of a certain pattern in a single widget as it is done by the request tracing page.

A configuration contains all pages. All changes made to a page or widget setup are contained in this configuration. To download this configuration as a JSON file select Export from the Settings top right menu.

To import a previously exported JSON file select Import in the top right Settings menu and select the JSON file to import.

This feature can be used to version and distribute configurations for similar installations.

The Monitoring console is built with the intention of being used to display metrics on status monitors. To allow viewing a variety of pages and their metrics on a single screen the pages can be automatically rotated showing each page for a configured interval duration.

To enable page rotation open the Settings and check the enabled checkbox in the Global Settings Page Rotation property. This starts the rotation immediately.

The duration can be adjusted changing the text field of the Page Rotation to another time value. Use units to express the duration, as for example: 1min, 30sec. Changes take effect immediately.

The rotation will show each page for the configured duration. Pages can be excluded from rotation by switching to the page using the page menu. Open the Settings and uncheck the Include in Rotation checkbox in the Page settings.

To create custom watches go to the Watches page using the top page menu or the Global settings.

The above example shows a relatively simple watch named HTTP server errors. The name of the watch can be anything as long as it is unique.

The watch should have both an Unhealthy and a Degraded level.

The field after the If holds the name of the metric series the watch is linked to, in this case the ns:http ServerCount5xx metric, as shown on the HTTP page. The in clause specifies the unit type of the given metric, here it is the Count of 5xx responses. Following is the comparison operator is selected, here >. This is followed by the threshold value, here 3 or 1, and the scope of the comparison, here in sample.

As both do not use a stop condition the checkbox after until is not checked.

The below table describes possible choices for each step of a watch condition.

The below table describes the semantics of the different types of comparison scopes.

After composing the watch conditions press Save or Update to install the watch. If a watch of the same name already exists the existing watch is replaced.

Watches that originate from server modules cannot be replaced. To change them disable the existing watch and click to edit it, this will create a duplicate with a different name that can be adjusted freely.

To edit a previously created watch click on its name in the list of watches or chose Edit from the cogs menu on the right.

Watches that originate from server modules cannot be edited. When clicking their names or when choosing Duplicate from their cogs menu a duplicate with a different name is created. The suggested name Copy of {original name} can be changed as long as it is different to existing watch names.

When changing the name of an edited watch to an existing name this overrides that watch. In other words watches are always identified by their name.

Go to the Watches page using the top page menu or the Global settings.

The status of each watch is reflected by the checkbox left of the watches name, checked for enabled, unchecked for disabled.

To disable an enabled watch un-check the checkbox or use the Disable item from the cogs menu on the right of the watch widget. To enable a disabled watch check the checkbox or use the Enable item from the cogs menu. The change has immediate effect.

The Core page gives an overview of some of the important instance metrics. Each instance in the grid is shown separately.

The HTTP page gives a rough overview of the state of the HTTP services of each instance.

The JVM pages gives a general overview of the state of the JVM running Payara Server.

A page that shows key metrics on the monitoring system itself that feeds the monitoring console with data.

This is mostly useful to identify problems with the monitoring system or to cross-check that unexpected metric values aren’t caused by general problems with monitoring.

A page that gives an overview of the state of the server health checks. The individual checks need to be enabled in the server HealthCheck configuration.

The Threads page gives detailed information on thread health of the cluster and lists information on stuck or hogging threads. Total number of live and daemon threads are given for reference.

The Request Tracing page overview page shows traces that exceeded the set threshold.

The Trace Duration Range widget shows all traces for the same method as a single bar. The length and position of the bar gives the range of total trace duration for all the occurrences that exceeded the threshold.

The bar starts at the minimum duration observed and ends at the maximum duration observed in the last minute. The legend shows the average duration and the method traced (innermost trace span operation)

When all time minimum and/or maximum are added in the Widget Data settings the bar starts and/or ends at these "all time" values instead of the minimum and/or maximum within the last minute.

The Trace Duration Above Threshold widget shows a graph where each point marks the duration of a "slow" request at the moment the requests occurred. Possible alerts are shown on top of the graph.

The Trace Data view adds tracing details for the "slow" requests that can be sorted by wall-time or value using the widget menu (gear icon in top right corner of the widget).

By clicking on one of the bars in the overview’s Trace Duration Range widget the details of all kept traces for this method can be viewed. This data corresponds to the traces stored in the trace-store.

For each trace, each of the spans corresponds to a bar in this graph.

By default, the traces are sorted longest (slowest) to shortest (fastest) total duration. On the time axis the spans are shown relative to the start of the trace.

When the side panel is opened the details of each trace span can be viewed by clicking on the span bar.

The legend gives average durations for each of the spans occurring for the traced method.

The spans can be sorted by wall time by selecting Sort by Wall Time or by duration by selecting Sort By Duration in the Sorting menu in the top right of the widget.

To get back to the overview close the details by clicking the x-icon on the right side of the Sorting menu.

The SQL page gives insight into slow SQL queries for all connection pools configured in Payara Server.

The main use of the Alerts page is to check for any server alerts. The page is split into two lists: the upper list containing all ongoing alerts, and the lower list containing past alerts.

As always for alert lists both lists are sorted starting with the most recent severe alert progressing to the oldest least severe alert. Annotations associated with alerts are hidden in this view to make room for more alert entries.

This page uses a query to populate the page with widgets for series that originate from the MicroProfile Metrics of the server and applications deployed on the server.

As this potentially are very many series the page by default is limited to at most 32 of them. The automatic placement will choose those series that show most dynamic. By default, the widget placement is updated every minute. This means the order and selection of widgets placed on the page can change once a minute based on their most recent dynamics.

The defaults can be customised using the Page settings.

Like the application metrics page this page uses a query to populate the page with widgets automatically.

The metric series originate from the MicroProfile Health checks of the server and any of the deployed applications. The purpose of this page is to give a quick strong visual indication of the health status of the cluster based on MicroProfile Health checks.

Each MicroProfile Health check is represented as a widget which shows the current status of the check in red (down for a while), amber (just went down) or green (up) for each of the instances where the check is performed.

In the example these are the checks Service1 and Service2 of the health-demo-1.0-SNAPSHOT application.

In addition to the MicroProfile Health checks there are aggregate metric series for each type (Health, Liveness, Readiness) per application and for the cluster as well as an aggregate Overall combining the three types health, liveness, readiness into a single metric series per application and for the cluster.