OpenTelemetry and OpenTracing support

Payara Server and Payara Micro natively support OpenTelemetry APIs as well as export of traces over OTLP protocol. Natively in this context means, that deployment of an OpenTelemetry Agent is not necessary and may lead to duplicate traces.

All OpenTracing support in Payara products is done by means of OpenTracing Compatibility Layer.

Payara Server and Payara Micro satisfy requirements of both MicroProfile OpenTracing 3.0 and MicroProfile Telemetry Tracing 1.1 provided appropriate Span Convention Settings are applied.

Implementation only covers the tracing aspect of OpenTelemetry, no metrics or logs are provided by default. Manual instrumentation is needed to enable these.

Enabling OpenTelemetry

OpenTelemetry is enabled for a particular application when:

  • It defines MicroProfile config property otel.sdk.disabled with value of false as defined by MicroProfile Telemetry Tracing

  • Request Tracing Service is enabled

  • System property otel.sdk.disabled is set to false

  • Environment variable OTEL_SDK_DISABLED is set to false

Configuring OpenTelemetry

When OpenTelemetry is enabled it proceeds to configure OpenTelemetry components by means of autoconfiguration, using the application’s MicroProfile Config properties prefixed with otel. first, but also falling back to system properties and environment variables as described in the linked documentation.

If the configured component does not ship with Payara distribution, it can be added as a server or application library as described in Using additional components.

Default Settings

With no other configuration properties specified OpenTelemetry components will:

  • Accept and propagate trace headers and baggage according to W3C recommendations Trace Context and Baggage

  • Export traces via OTLP protocol to http://localhost:4317

  • Respect parent trace’s sampling decision and always sample local root traces (sampler setting parentbased_always_on)

  • Will not export metrics or logs

To easily receive traces in default configuration run the following Jaeger container and then access its UI at port 16686:

docker run --name jaeger \
   -e COLLECTOR_OTLP_ENABLED=true \
   -p 16686:16686 \
   -p 4317:4317 \
   -p 4318:4318 \
   jaegertracing/all-in-one:latest

Span Convention Settings

In order to improve consistency among products our implementation offers several strategies of naming and tagging spans of Jakarta REST client and server calls. These are configured by specifying the MicroProfile Config property payara.telemetry.span-convention for your application. The following values are accepted:

  • opentelemetry follows convention prescribed by OpenTelemetry semantic conventions for HTTP spans (default setting)

  • opentracing-http-path satisfies MicroProfile OpenTracing setting for Operation Name Provider http-path

  • opentracing-class-method corresponds to MicroProfile OpenTracing setting for Operation Name Provider class-method

  • microprofile-telemetry satisfies span naming as tested by the MicroProfile Telemetry Tracing TCK

Span Name Examples

Let’s assume GET request made to /app/rest/resource/method, where

  • /app is application context root

  • /rest is @ApplicationPath of Jakarta REST

  • /resource is @Path annotation of class example.Resource

  • /method is @Path annotation of method restMethod

The following span names will be created for this request for different Span Convention settings:

span-convention setting Server Span Name

opentelemetry

GET /app/rest/resource/method

opentracing-class-method

GET:example.Resource.restMethod

opentracing-http-path

GET:/resource/method

microprofile-telemetry

/app/rest/resource/method

Suppressing Export Warning Message

When the export endpoint is not available in default setting the following error message is logged by logger io.opentelemetry.exported.internal.grpc.OkHttpGrpcExporter:

Failed to export spans. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4317

If you don’t intend to run an OTEL endpoint on localhost or at all you need to configure exporter by means of MP Config Properties, system properties, or environment variables:

  • in order to turn off exports set otel.traces.exporter to none

  • in order to set remote endpoint otel.exporter.otlp.endpoint to http://<collector-host>:<port>;

Manual tracing

In order to fine-tune spans and have better control about their boundaries one can use directly either the OpenTracing or OpenTelemetry APIs.

Dependencies

The appropriate versions of APIs are provided by Payara BOM artifact matching the version of Payara Server or Micro you are using.

Payara Platform provides the following APIs and it is advised to mark them as provided dependencies in your application and don’t package them with it:

  • io.opentelemetry:opentelemetry-sdk-extension-autoconfigure

  • io.opentelemetry:opentelemetry-sdk

  • io.opentelemetry:opentelemetry-sdk-common

  • io.opentelemetry:opentelemetry-sdk-metrics

  • io.opentelemetry:opentelemetry-sdk-logs

  • io.opentelemetry:opentelemetry-api-logs

  • io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi

  • io.opentelemetry:opentelemetry-semconv

  • io.opentelemetry:opentelemetry-opentracing-shim

  • io.opentelemetry:opentelemetry-api

  • io.opentelemetry:opentelemetry-context

  • io.opentracing:opentracing-api

  • io.opentracing:opentracing-noop

  • io.opentelemetry:opentelemetry-exporter-otlp

  • io.opentelemetry:opentelemetry-sdk-trace

  • io.opentelemetry:opentelemetry-exporter-otlp-common

  • io.opentelemetry:opentelemetry-exporter-common

Using OpenTelemetry Manually

import io.opentelemetry.api.trace.Tracer;

@RequestScoped
public class TracedComponent {

    @Inject
    Tracer tracer;

    public void tracedMethod() {
        var span = tracer.spanBuilder("tracedTask").startSpan();

        try (var scope = span.makeCurrent()) {
            // do work, add information to span

            if (span.isRecording()) {
                // compute expensive events or tags
                // only if the span is being sampled
            }
        } finally {
            span.setStatus(StatusCode.OK);
            span.end();
        }
    }
}

Using OpenTracing Manually

import io.opentracing.Tracer;

@RequestScoped
public class TracedComponent {

    @Inject
    Tracer tracer;

    public void tracedMethod() {
        var span = tracer.buildSpan("tracedTask").start();

        try (var scope = tracer.activateSpan(span)) {
            // do work, add information to span
        } finally {
            span.finish();
        }
    }
}

Using additional components

If your application requires OpenTracing components that are not shipped with Payara Platform, it is possible to either put them in common libraries, or ship them with an application. Extension components can be even coded directly in the application code.

Provider components in library directory

Autoconfiguration can pick up components, which are placed in directory <domaindir>/lib for example by means of command asadmin add-library.

Example: adding export to log

  1. Download opentelemetry-exporter-logging.jar

  2. Run asadmin add-library opentelemetry-exporter-logging.jar

  3. Use the component by defining otel.traces.exporter=logging

Provider components in application code

Applications can declare and use their own OpenTelemetry components such as exporters or samplers by writing against autoconfiguration SPI and placing appropriate Service Loader resource in the applicaiton

Example: Minimal logging exporter

src/main/resources/META-INF/services/io.opentelemetry.sdk.autoconfigure.spi.traces.ConfigurableSpanExporterProvider
example.LogExporter.Provider
src/main/java/example/LogExporter.java
public class LogExporter implements SpanExporter {

    private static final Logger LOGGER = Logger.getLogger(LogExporter.class.getName());

    @Override
    public CompletableResultCode export(Collection<SpanData> spans) {
        spans.forEach(s -> LOGGER.info(s.toString()));
        return CompletableResultCode.ofSuccess();
    }

    @Override
    public CompletableResultCode flush() {
        return CompletableResultCode.ofSuccess();
    }

    @Override
    public CompletableResultCode shutdown() {
        return CompletableResultCode.ofSuccess();
    }

    // This is registered as SPI and creates configured exporter
    public static class Provider implements ConfigurableSpanExporterProvider {

        @Override
        public SpanExporter createExporter(ConfigProperties configProperties) {
            return new LogExporter();
        }

        @Override
        public String getName() {
            return "logs";
        }
    }
}

Relation to Request Tracing Service

All spans created by OpenTelemetry and OpenTracing APIs are passed into Request Tracing Service when they end. Propagated trace IDs are propagated into Request Tracing Service at the start of spans.

Sampling decisions of Request Tracing service are not taken into account when these traces are created, because Request Tracing Service does tracing decision at the end of span rather than at the beginning. This means that traces might get exported via OTLP but not passed to the notifiers configured in the Request Tracing Service, because the service might have taken the decision not to sample this particular trace.

Information that server’s components put into Request Tracing Service is not available to OpenTelemetry traces, as information does not flow in this direction. However, more and more parts of Payara Server will gradually switch to using OpenTelemetry API natively and therefore this gap will eventually close.