Custom Vendor Metrics

The Payara Platform supports exporting the JMX Beans and HealthCheck stats to the Vendor Metrics.

JMX MBeans

JMX MBeans can be exposed as Custom Vendor Metrics by the user by supplying a custom metrics.xml file in the domain-dir/config/ directory with the following format:

<config>
    <base>
        ...
    </base>
    <vendor>
        <!--sample metadata-->
        <metadata>
            <tags>
                <tag>
                    <name>operatingSystem</name>
                    <value>Linux</value>
                </tag>
            </tags>
            <name>system.cpu.load</name>
            <mbean>java.lang:type=OperatingSystem/SystemCpuLoad</mbean>
            <type>gauge</type>
            <unit>none</unit>
            <displayName>System CPU Load</displayName>
            <description>Recent CPU usage for the whole system.</description>
        </metadata>
    </vendor>
</config>
Payara Server has a more extensive range of MBeans available than Payara Micro, but these AMX MBeans need to be enabled through the Monitoring section of the admin console before they will be usable.
Each of these AMX MBeans are also lazily loaded so, for example, JDBC connection pool MBeans will not be visible until an application that uses the connection pool is deployed.

Templating Metrics for AMX MBeans

It is possible to create a template for a metric based on AMX MBeans properties for multiple resource configurations or similar objects (like JDBC connection pools for example).

Simple Templating

You can use the %s placeholder to define a template in the metadata definition of the metric using the mbean element. The server will look for all MBeans that match the expression defined and will generate metrics for each one of them.

You will need to use the %s placeholder in the name element as well to allow each metric to have a different name and avoid collisions.
Example

Here’s a simple example of defining a custom metric definition template that will generate metrics for the number of free connections available to all JDBC connections pools:

<metadata>
     <tags>
        <tag>
            <name>name</name>
            <value>%s</value>
        </tag>
     </tags>
     <name>jdbc.connection.pool.numconnfree</name>
     <mbean>amx:pp=/mon/server-mon[server],type=jdbc-connection-pool-mon,name=resources/%sPool/numconnfree#current</mbean>
     <type>counter</type>
     <unit>none</unit>
     <displayName>numconnfreePool</displayName>
     <description>The total number of free connections in the pool as of the last sampling.</description>
</metadata>

Which will yield the following results:

# TYPE vendor_jdbc_connection_pool_numconnfree_total counter
# HELP vendor_jdbc_connection_pool_numconnfree_total The total number of free connections in the pool as of the last sampling.
vendor_jdbc_connection_pool_numconnfree_total{name="resources/DerbyPool"} 0
vendor_jdbc_connection_pool_numconnfree_total{name="resources/H2Pool"} 0
vendor_jdbc_connection_pool_numconnfree_total{name="resources/__TimerPool"} 0 0
Instance Names

Payara provides the placeholder ${instance}. It is replaced with the name of the Payara Server instance, where the metrics is running:

<metadata>
    <name>requestcount.${instance}</name>
    <mbean>amx:type=web-request-mon,pp=/mon/server-mon[${instance}],name=clusterjsp/server/requestcount#count</mbean>
    <type>gauge</type>
    <unit>none</unit>
    <displayName>Request count for application</displayName>
</metadata>

Here’s the sample output of this metric being queried on a server instance named i1:

vendor_requestcount_i1 1

Advanced Templating

If you need more control on the attributes of specific MBeans used in a metric definition, you can use the ${key}, ${attribute} and ${subattribute} placeholders to define more fine-grained metrics.

These placeholders are used as follows:

key

Used as a placeholder of the name (or part of the name) of an AMX MBean.

attribute

Used as a placeholder for the name of an attribute of an AMX MBean.

subattribute

Used as a placeholder for the name of a sub-attribute (or second-level attribute), which corresponds to the attribute of a CompositeData object.

You will need to use the ${key}, ${attribute} and ${subattribute} placeholders in the name element as well to prevent name collisions between different templated metrics.
Example

Here’s an example of defining a custom metric definition template with these placeholders that will generate a counter metric for each specific connection metric that is available to all JDBC connection pools:

<metadata>
     <name>jdbc.connection.pool.${key}Pool.${attribute}#${subattribute}</name>
     <mbean>amx:pp=/mon/server-mon[server],type=jdbc-connection-pool-mon,name=resources/${key}Pool/${attribute}#${subattribute}</mbean>
     <type>counter</type>
     <unit>none</unit>
</metadata>

Which will yield the following results:

# TYPE vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconnsuccessfullymatched#start_time counter
vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconnsuccessfullymatched#start_time 1540463722554
# TYPE vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconncreated#count counter
vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconncreated#count 0
# TYPE vendor:jdbc_connection_pool_resources/_derby_pool_pool_connrequestwaittime#last_sample_time counter
vendor:jdbc_connection_pool_resources/_derby_pool_pool_connrequestwaittime#last_sample_time -1
# TYPE vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconnused#start_time counter
vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconnused#start_time 1540463106138
# TYPE vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconnused#last_sample_time counter
vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconnused#last_sample_time 1540463722554
# TYPE vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconntimedout#start_time counter
vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconntimedout#start_time 1540463722554
# TYPE vendor:jdbc_connection_pool_resources/_derby_pool_pool_connrequestwaittime#start_time counter
vendor:jdbc_connection_pool_resources/_derby_pool_pool_connrequestwaittime#start_time 1540463722554
# TYPE vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconnfree#start_time counter
vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconnfree#start_time 1540463106138
# TYPE vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconnfailedvalidation#count counter
vendor:jdbc_connection_pool_resources/_derby_pool_pool_numconnfailedvalidation#count 0
......
# TYPE vendor:jdbc_connection_pool_resources/_h2_pool_pool_numconnsuccessfullymatched#start_time counter
vendor:jdbc_connection_pool_resources/_h2_pool_pool_numconnsuccessfullymatched#start_time 1540463722554
# TYPE vendor:jdbc_connection_pool_resources/_h2_pool_pool_numconncreated#count counter
vendor:jdbc_connection_pool_resources/_h2_pool_pool_numconncreated#count 0
......
......

HealthCheck Stats

HealthCheck Services can be exposed as Custom Vendor Metrics by the user by supplying a custom metrics.xml file in the domain-dir/config/ directory as well.

The following statistics are the integrated HealthCheck services metrics that are automatically exposed to the Microprofile Metrics service:

HealthCheck Service Service ID Attributes Sub-Attributes

Stuck Threads

healthcheck-stuck

NA

count, maxDuration

Connection Pool

healthcheck-cpool

by default H2Pool, __TimerPool

usedConnection, freeConnection, totalConnection

Example

Here’s an example of exposing the HealthCheck stats to the Microprofile Metrics endpoint:

<metadata>
    <name>thread.stuck.count</name>
    <service>healthcheck-stuck#count</service>
    <type>gauge</type>
    <unit>none</unit>
    <displayName>Stuck Thread Count</displayName>
    <description>Displays the stuck thread count which is blocked, and can't return to the thread pool for a certain amount of time.</description>
</metadata>

Which will yield the following results:

# TYPE vendor_thread_stuck_count gauge
# HELP vendor_thread_stuck_count Displays the stuck thread count which is blocked, and can't return to the threadpool for a certain amount of time.
vendor_thread_stuck_count 0
......
......
Health Check services need to be enabled first

Templating Metrics for HealthCheck Service

It is possible to create a template for a metric for multiple resource, attributes or sub-attributes configurations.

These placeholders are used as follows:

attribute

Used as a placeholder for the name of an attribute of an AMX MBean.

subattribute

Used as a placeholder for the name of a sub-attribute (or second-level attribute), which corresponds to the attribute of a CompositeData object.

You will need to use the ${attribute} and ${subattribute} placeholders in the name element as well to prevent name collisions between different templated metrics.

Example

Here’s an example of defining a custom metric definition template with these placeholders that will generate a counter metric for each specific connection metric that is available to all JDBC connection pools:

<metadata>
    <name>connection.pool.${attribute}.${subattribute}</name>
    <service>healthcheck-cpool/${attribute}#${subattribute}</service>
    <type>counter</type>
    <unit>none</unit>
    <displayName>${attribute} ${subattribute}</displayName>
    <description>Displays the number of ${subattribute} in the Connection Pool ${attribute}.</description>
</metadata>

Which will yield the following results:

# TYPE vendor_connection_pool_H2Pool_freeConnection_total counter
# HELP vendor_connection_pool_H2Pool_freeConnection_total Displays the number of freeConnection in the Connection Pool H2Pool.
vendor_connection_pool_H2Pool_freeConnection_total 0
# TYPE vendor_connection_pool_H2Pool_totalConnection_total counter
# HELP vendor_connection_pool_H2Pool_totalConnection_total Displays the number of totalConnection in the Connection Pool H2Pool.
vendor_connection_pool_H2Pool_totalConnection_total 0
# TYPE vendor_connection_pool_H2Pool_usedConnection_total counter
# HELP vendor_connection_pool_H2Pool_usedConnection_total Displays the number of usedConnection in the Connection Pool H2Pool.
vendor_connection_pool_H2Pool_usedConnection_total 0
# TYPE vendor_connection_pool___TimerPool_freeConnection_total counter
# HELP vendor_connection_pool___TimerPool_freeConnection_total Displays the number of freeConnection in the Connection Pool __TimerPool.
vendor_connection_pool___TimerPool_freeConnection_total 0
# TYPE vendor_connection_pool___TimerPool_totalConnection_total counter
# HELP vendor_connection_pool___TimerPool_totalConnection_total Displays the number of totalConnection in the Connection Pool __TimerPool.
vendor_connection_pool___TimerPool_totalConnection_total 0
# TYPE vendor_connection_pool___TimerPool_usedConnection_total counter
# HELP vendor_connection_pool___TimerPool_usedConnection_total Displays the number of usedConnection in the Connection Pool __TimerPool.
vendor_connection_pool___TimerPool_usedConnection_total 0
......
......