OpenTelemetry and OpenTracing Support

The Payara Platform natively supports OpenTelemetry APIs as well as export of traces over the OTLP protocol. Natively in this context means, that deployment of an OpenTelemetry Agent is not necessary and may lead to duplicate traces.

All OpenTracing support in the Payara Platform is done by means of the OpenTracing Compatibility Layer.

The Payara Platform satisfies requirements of both MicroProfile OpenTracing 3.0 and MicroProfile Telemetry Tracing 1.0 provided appropriate Span Convention Settings are applied.

OpenTelemetry implementation only covers the tracing aspect of OpenTelemetry, no metrics or logs are provided by default.
Manual instrumentation is needed to enable these.

Enabling OpenTelemetry

OpenTelemetry is enabled for a particular application when:

  • It defines the MicroProfile configuration property otel.sdk.disabled with a value of false as defined by MicroProfile Telemetry Tracing

  • The Request Tracing Service is enabled

  • The otel.sdk.disabled System property is set to false

  • The OTEL_SDK_DISABLED Environment variable is set to false

Configuring OpenTelemetry

When OpenTelemetry is enabled it proceeds to configure OpenTelemetry components by means of autoconfiguration, using the application’s MicroProfile Config properties prefixed with otel. first, but also falling back to system properties and environment variables as described in the linked documentation.

If the configured component does not ship with Payara distribution, it can be added as a server or application library as described in Using additional components.

Default Settings

With no other configuration properties specified OpenTelemetry components will:

  • Accept and propagate trace headers and baggage according to W3C recommendations Trace Context and Baggage

  • Export traces via OTLP protocol to http://localhost:4317

  • Respect parent trace’s sampling decision and always sample local root traces (sampler setting parentbased_always_on)

  • Will not export metrics or logs.

To easily receive traces in default configuration run the following Jaeger container and then access its UI at port 16686:

docker run --name jaeger \
   -e COLLECTOR_OTLP_ENABLED=true \
   -p 16686:16686 \
   -p 4317:4317 \
   -p 4318:4318 \
   jaegertracing/all-in-one:latest

Span Convention Settings

In order to improve consistency among products our implementation offers several strategies of naming and tagging spans of Jakarta REST client and server calls.

These are configured by specifying the MicroProfile Config property payara.telemetry.span-convention for your application.

The following values are accepted:

  • opentelemetry follows convention prescribed by OpenTelemetry semantic conventions for HTTP spans (default setting)

  • opentracing-http-path satisfies MicroProfile OpenTracing setting for Operation Name Provider http-path

  • opentracing-class-method corresponds to MicroProfile OpenTracing setting for Operation Name Provider class-method

  • microprofile-telemetry satisfies span naming as tested by the MicroProfile Telemetry Tracing TCK

Span Name Examples

Let’s assume GET request made to /app/rest/resource/method, where

  • /app is application context root

  • /rest is @ApplicationPath of Jakarta REST

  • /resource is @Path annotation of class example.Resource

  • /method is @Path annotation of method restMethod

The following span names will be created for this request for different Span Convention settings:

span-convention setting Server Span Name

opentelemetry

GET /app/rest/resource/method

opentracing-class-method

GET:example.Resource.restMethod

opentracing-http-path

GET:/resource/method

microprofile-telemetry

/app/rest/resource/method

Suppressing Export Warning Message

When the export endpoint is not available in default setting the following error message is logged by logger io.opentelemetry.exported.internal.grpc.OkHttpGrpcExporter:

Failed to export spans. The request could not be executed. Full error message: Failed to connect to `localhost/127.0.0.1:4317`

If you don’t intend to run an OTEL endpoint on localhost or at all you need to configure the exporter by means of MP Config Properties, system properties, or environment variables:

  • To turn off exports, set the otel.traces.exporter property to none

  • To set the remote endpoint otel.exporter.otlp.endpoint to http://<collector-host>:<port>;

Manual Tracing

To fine-tune spans and have better control about their boundaries you can use directly either the OpenTracing or OpenTelemetry APIs.

Dependencies

The appropriate versions of APIs are provided by Payara BOM artifact matching the version of the Payara Platform distribution you are using.

Payara Platform provides the following APIs, and it is advised to mark them as provided dependencies in your application and don’t package them with it:

  • io.opentelemetry:opentelemetry-sdk-extension-autoconfigure

  • io.opentelemetry:opentelemetry-sdk

  • io.opentelemetry:opentelemetry-sdk-common

  • io.opentelemetry:opentelemetry-sdk-metrics

  • io.opentelemetry:opentelemetry-sdk-logs

  • io.opentelemetry:opentelemetry-api-logs

  • io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi

  • io.opentelemetry:opentelemetry-semconv

  • io.opentelemetry:opentelemetry-opentracing-shim

  • io.opentelemetry:opentelemetry-api

  • io.opentelemetry:opentelemetry-context

  • io.opentracing:opentracing-api

  • io.opentracing:opentracing-noop

  • io.opentelemetry:opentelemetry-exporter-otlp

  • io.opentelemetry:opentelemetry-sdk-trace

  • io.opentelemetry:opentelemetry-exporter-otlp-common

  • io.opentelemetry:opentelemetry-exporter-common

Using OpenTelemetry Manually

import io.opentelemetry.api.trace.Tracer;

@RequestScoped
public class TracedComponent {

    @Inject
    Tracer tracer;

    public void tracedMethod() {
        var span = tracer.spanBuilder("tracedTask").startSpan();

        try (var scope = span.makeCurrent()) {
            // do work, add information to span
            if (span.isRecording()) {
                // compute expensive events or tags
                // only if the span is being sampled
            }
        } finally {
            span.setStatus(StatusCode.OK);
            span.end();
        }
    }
}

Using OpenTracing Manually

import io.opentracing.Tracer;

@RequestScoped
public class TracedComponent {

    @Inject
    Tracer tracer;

    public void tracedMethod() {
        var span = tracer.buildSpan("tracedTask").start();

        try (var scope = tracer.activateSpan(span)) {
            // do work, add information to span
        } finally {
            span.finish();
        }
    }
}

Using additional components

If your application requires OpenTracing components that are not shipped with the Payara Platform, it is possible to either put them in common libraries, or ship them with an application. Extension components can be even coded directly in the application code.

Provider components in library directory

Applies to Payara Server only.

Autoconfiguration can pick up components, which are placed in directory domain_dir/lib for example by means of the asadmin add-library command.

Example: adding export to log

  1. Download opentelemetry-exporter-logging.jar

  2. Run asadmin add-library opentelemetry-exporter-logging.jar

  3. Use the component by defining otel.traces.exporter=logging

Provider components in application code

Applications can declare and use their own OpenTelemetry components such as exporters or samplers by writing against autoconfiguration SPI and placing appropriate Service Loader resource in the application

Example: Minimal logging exporter

src/main/resources/META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.traces.ConfigurableSpanExporterProvider
fish.payara.example.LogExporter.Provider
src/main/java/fish/payara/example/LogExporter.java
public class LogExporter implements SpanExporter {

    private static final Logger LOGGER = Logger.getLogger(LogExporter.class.getName());

    @Override
    public CompletableResultCode export(Collection<SpanData> spans) {
        spans.forEach(s -> LOGGER.info(s.toString()));
        return CompletableResultCode.ofSuccess();
    }

    @Override
    public CompletableResultCode flush() {
        return CompletableResultCode.ofSuccess();
    }

    @Override
    public CompletableResultCode shutdown() {
        return CompletableResultCode.ofSuccess();
    }

    // This is registered as SPI and creates configured exporter
    public static class Provider implements ConfigurableSpanExporterProvider {

        @Override
        public SpanExporter createExporter(ConfigProperties configProperties) {
            return new LogExporter();
        }

        @Override
        public String getName() {
            return "logs";
        }
    }
}

Relation to Request Tracing Service

All spans created by OpenTelemetry and OpenTracing APIs are passed into Request Tracing Service when they end. Propagated trace IDs are propagated into Request Tracing Service at the start of spans.

Sampling decisions of Request Tracing service are not taken into account when these traces are created, because the Request Tracing Service makes a tracing decision at the end of span rather than at the beginning.

This means that traces might get exported via OTLP but not passed to the notifiers configured in the Request Tracing Service, because the service might have taken the decision not to sample this particular trace.

Information that server’s components put into Request Tracing Service is not available to OpenTelemetry traces, as information does not flow in this direction.

However, more and more parts of Payara Server will gradually switch to using OpenTelemetry API natively and therefore this gap will eventually close.