OpenTelemetry and OpenTracing Support
The Payara Platform natively supports OpenTelemetry APIs as well as export of traces over the OTLP protocol. Natively in this context means, that deployment of an OpenTelemetry Agent is not necessary and may lead to duplicate traces.
All OpenTracing support in the Payara Platform is done by means of the OpenTracing Compatibility Layer.
The Payara Platform satisfies requirements of both MicroProfile OpenTracing 3.0 and MicroProfile Telemetry Tracing 1.1 provided appropriate Span Convention Settings are applied.
OpenTelemetry implementation only covers the tracing aspect of OpenTelemetry, no metrics or logs are provided by default. Manual instrumentation is needed to enable these. |
Enabling OpenTelemetry
OpenTelemetry is enabled for a particular application when:
-
It defines the MicroProfile configuration property
otel.sdk.disabled
with a value offalse
as defined by MicroProfile Telemetry Tracing -
The Request Tracing Service is enabled
-
The
otel.sdk.disabled
System property is set tofalse
-
The
OTEL_SDK_DISABLED
Environment variable is set tofalse
Configuring OpenTelemetry
When OpenTelemetry is enabled it proceeds to configure OpenTelemetry components by means of autoconfiguration, using the application’s MicroProfile Config properties prefixed with otel.
first, but also falling back to system properties and environment variables as described in the linked documentation.
If the configured component does not ship with Payara distribution, it can be added as a server or application library as described in Using additional components.
Default Settings
With no other configuration properties specified OpenTelemetry components will:
-
Accept and propagate trace headers and baggage according to W3C recommendations Trace Context and Baggage
-
Export traces via OTLP protocol to
http://localhost:4317
-
Respect parent trace’s sampling decision and always sample local root traces (sampler setting
parentbased_always_on
) -
Will not export metrics or logs.
To easily receive traces in default configuration run the following Jaeger container and then access its UI at port 16686
:
docker run --name jaeger \
-e COLLECTOR_OTLP_ENABLED=true \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
jaegertracing/all-in-one:latest
Span Convention Settings
In order to improve consistency among products our implementation offers several strategies of naming and tagging spans of Jakarta REST client and server calls.
These are configured by specifying the MicroProfile Config property payara.telemetry.span-convention
for your application.
The following values are accepted:
-
opentelemetry
follows convention prescribed by OpenTelemetry semantic conventions for HTTP spans (default setting) -
opentracing-http-path
satisfies MicroProfile OpenTracing setting for Operation Name Providerhttp-path
-
opentracing-class-method
corresponds to MicroProfile OpenTracing setting for Operation Name Providerclass-method
-
microprofile-telemetry
satisfies span naming as tested by the MicroProfile Telemetry Tracing TCK
Span Name Examples
Let’s assume GET
request made to /app/rest/resource/method
, where
-
/app
is application context root -
/rest
is@ApplicationPath
of Jakarta REST -
/resource
is@Path
annotation of classexample.Resource
-
/method
is@Path
annotation of methodrestMethod
The following span names will be created for this request for different Span Convention settings:
span-convention setting | Server Span Name |
---|---|
|
|
|
|
|
|
|
|
Suppressing Export Warning Message
When the export endpoint is not available in default setting the following error message is logged by logger io.opentelemetry.exported.internal.grpc.OkHttpGrpcExporter
:
Failed to export spans. The request could not be executed. Full error message: Failed to connect to `localhost/127.0.0.1:4317`
If you don’t intend to run an OTEL endpoint on localhost
or at all you need to configure the exporter by means of MP Config Properties, system properties, or environment variables:
-
To turn off exports, set the
otel.traces.exporter
property tonone
-
To set the remote endpoint
otel.exporter.otlp.endpoint
tohttp://<collector-host>:<port>
Manual Tracing
To fine-tune spans and have better control about their boundaries you can use directly either the OpenTracing or OpenTelemetry APIs.
Dependencies
The appropriate versions of APIs are provided by Payara BOM artifact matching the version of the Payara Platform distribution you are using.
Payara Platform provides the following APIs, and it is advised to mark them as provided
dependencies in your application and don’t package them with it:
-
io.opentelemetry:opentelemetry-sdk-extension-autoconfigure
-
io.opentelemetry:opentelemetry-sdk
-
io.opentelemetry:opentelemetry-sdk-common
-
io.opentelemetry:opentelemetry-sdk-metrics
-
io.opentelemetry:opentelemetry-sdk-logs
-
io.opentelemetry:opentelemetry-api-logs
-
io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi
-
io.opentelemetry:opentelemetry-semconv
-
io.opentelemetry:opentelemetry-opentracing-shim
-
io.opentelemetry:opentelemetry-api
-
io.opentelemetry:opentelemetry-context
-
io.opentracing:opentracing-api
-
io.opentracing:opentracing-noop
-
io.opentelemetry:opentelemetry-exporter-otlp
-
io.opentelemetry:opentelemetry-sdk-trace
-
io.opentelemetry:opentelemetry-exporter-otlp-common
-
io.opentelemetry:opentelemetry-exporter-common
Using OpenTelemetry Manually
import io.opentelemetry.api.trace.Tracer;
@RequestScoped
public class TracedComponent {
@Inject
Tracer tracer;
public void tracedMethod() {
var span = tracer.spanBuilder("tracedTask").startSpan();
try (var scope = span.makeCurrent()) {
// do work, add information to span
if (span.isRecording()) {
// compute expensive events or tags
// only if the span is being sampled
}
} finally {
span.setStatus(StatusCode.OK);
span.end();
}
}
}
Using OpenTracing Manually
import io.opentracing.Tracer;
@RequestScoped
public class TracedComponent {
@Inject
Tracer tracer;
public void tracedMethod() {
var span = tracer.buildSpan("tracedTask").start();
try (var scope = tracer.activateSpan(span)) {
// do work, add information to span
} finally {
span.finish();
}
}
}
Using additional components
If your application requires OpenTracing components that are not shipped with the Payara Platform, it is possible to either put them in common libraries, or ship them with an application. Extension components can be even coded directly in the application code.
Provider components in library directory
Applies to Payara Server only. |
Autoconfiguration can pick up components, which are placed in directory domain_dir/lib
for example by means of the asadmin add-library
command.
Example: adding export to log
-
Download opentelemetry-exporter-logging.jar
-
Run
asadmin add-library opentelemetry-exporter-logging.jar
-
Use the component by defining
otel.traces.exporter=logging
Provider components in application code
Applications can declare and use their own OpenTelemetry components such as exporters or samplers by writing against autoconfiguration SPI and placing appropriate Service Loader resource in the application
Example: Minimal logging exporter
fish.payara.example.LogExporter.Provider
public class LogExporter implements SpanExporter {
private static final Logger LOGGER = Logger.getLogger(LogExporter.class.getName());
@Override
public CompletableResultCode export(Collection<SpanData> spans) {
spans.forEach(s -> LOGGER.info(s.toString()));
return CompletableResultCode.ofSuccess();
}
@Override
public CompletableResultCode flush() {
return CompletableResultCode.ofSuccess();
}
@Override
public CompletableResultCode shutdown() {
return CompletableResultCode.ofSuccess();
}
// This is registered as SPI and creates configured exporter
public static class Provider implements ConfigurableSpanExporterProvider {
@Override
public SpanExporter createExporter(ConfigProperties configProperties) {
return new LogExporter();
}
@Override
public String getName() {
return "logs";
}
}
}
Relation to Request Tracing Service
All spans created by OpenTelemetry and OpenTracing APIs are passed into Request Tracing Service when they end. Propagated trace IDs are propagated into Request Tracing Service at the start of spans.
Sampling decisions of Request Tracing service are not taken into account when these traces are created, because the Request Tracing Service makes a tracing decision at the end of span rather than at the beginning.
This means that traces might get exported via OTLP but not passed to the notifiers configured in the Request Tracing Service, because the service might have taken the decision not to sample this particular trace.
Information that server’s components put into Request Tracing Service is not available to OpenTelemetry traces, as information does not flow in this direction.
However, more and more parts of Payara Server will gradually switch to using OpenTelemetry API natively and therefore this gap will eventually close.