Observability¶

Agenor ships with a thin, dependency-free telemetry abstraction (AgenorTelemetry) in agenor-core. By default, every instrumented component uses the built-in no-op implementation, which has zero overhead and introduces no external dependencies. The real OpenTelemetry SDK integration lives in agenor-adapters and is entirely opt-in.

Opting in to OpenTelemetry¶

1 — Add the OTel dependency¶

agenor-adapters declares opentelemetry-sdk as an optional dependency (per ADR-018). You must explicitly pull it in your own pom.xml:

<!-- Your application POM -->
<dependencies>
    <dependency>
        <groupId>dev.agenor</groupId>
        <artifactId>agenor-adapters</artifactId>
    </dependency>

    <!-- Opt-in: OTel SDK + OTLP exporter -->
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-sdk</artifactId>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-otlp</artifactId>
    </dependency>
</dependencies>

Consumers who do not add these dependencies compile and run cleanly — no ClassNotFoundException at runtime.

2a — Spring Boot applications (auto-configuration)¶

When io.opentelemetry.api.OpenTelemetry is on the classpath, the Spring Boot starter auto-configures OtelAgenorTelemetry via @ConditionalOnClass. Add the following to application.yml:

agenor:
  telemetry:
    enabled: true
    service-name: my-agent-service
    exporter: otlp-http           # otlp-http | otlp-grpc | none
    endpoint: http://localhost:4318

2b — Manual wiring (no Spring)¶

import dev.agenor.adapters.telemetry.OtelTelemetryFactory;
import dev.agenor.core.telemetry.AgenorTelemetry;

AgenorTelemetry telemetry = OtelTelemetryFactory.builder()
        .serviceName("my-agent-service")
        .otlpHttpExporter("http://localhost:4318")
        .build();

AgenorRuntime runtime = AgenorRuntime.builder()
        .telemetry(telemetry)
        .build();

Span taxonomy¶

The table below lists every span emitted by Agenor components. Spans marked Redis adapter are only emitted when the Redis messaging backend is active (agenor.messaging.provider: redis).

Span name	Component	Key attributes
`llm.chat`	`InstrumentedLLMProvider`	`llm.provider`, `llm.model`, `llm.tokens.input`, `llm.tokens.output`, `llm.latency_ms`
`llm.chat.stream`	`InstrumentedLLMProvider`	same as `llm.chat` + `llm.stream.chunks`
`guardrail.evaluate`	`GuardrailChain`	`guardrail.name`, `guardrail.direction` (`input`\|`output`), `guardrail.decision` (`passed`\|`blocked`)
`hitl.approval`	`HumanCheckpointBehavior`	`hitl.request_id`, `hitl.action`, `hitl.decision`, `hitl.wait_ms`
`behavior.execute`	`SimpleBehaviorScheduler`	`behavior.id`, `behavior.type`, `agent.id`, `behavior.duration_ms`
`mcp.tool.call`	`AgenorMcpClientAdapter`	`mcp.tool.name`, `mcp.transport` (`sse`\|`stdio`)
`message.send`	`InMemoryMessageDispatcher`	`message.topic` or `message.recipient`, `message.id`, `agent.sender`
`agent.receive`	`DefaultAgentMailbox`	`agent.id`, `message.id`, `message.topic`, `agent.sender`, `message.correlation_id`, `conversation.id`, `mailbox.lane` (`dialogue`\|`push`)
`directory.resolve`	`InMemoryAgentDirectory`, `JdbcAgentResolver` (JDBC adapter)	`agent.id`, `endpoint.type` (`not-found` if missing)
`directory.register`	`JdbcAgentRegistry` (JDBC adapter)	`agent.id`
`directory.unregister`	`JdbcAgentRegistry` (JDBC adapter)	`agent.id`
`directory.update_status`	`JdbcAgentRegistry` (JDBC adapter)	`agent.id`, `agent.status`
`directory.find`	`JdbcAgentDiscovery` (JDBC adapter)	`directory.find.type` (`by_id`\|`by_capability`\|`by_type`\|`query`), `directory.find.result_count`
`message.publish`	`RedisTopicPublisher` (Redis adapter)	`message.topic`, `message.id`, `agent.sender`, `transport.type`
`transport.send`	`RedisMessageTransport` (Redis adapter)	`transport.type`, `transport.endpoint`, `message.id`, `agent.sender`
`message.receive`	`ConsumerLoop` (Redis adapter)	`message.id`, `message.topic`, `agent.sender`, `message.correlation_id`, `transport.type`

All spans use SpanStatus.OK on success and SpanStatus.ERROR on exception. Exceptions are recorded via Span.recordException(Throwable).

The message.correlation_id attribute on message.receive spans carries the same value set by the publisher, enabling correlation between message.publish and message.receive spans in your APM tool even without native OTel span links.

`agent.receive` is the transport-independent one¶

message.receive is emitted by the Redis adapter's consumer loop, so it exists only on that transport: run the same agent on the in-memory dispatcher and nothing was traced on the receive side at all. Whether an arriving message showed up in your traces therefore depended on where it came from.

agent.receive is emitted by the mailbox, which every inbound message goes through whatever delivered it. On Redis the two nest — the transport's hop around the agent's handling — which is why they carry different names rather than one name meaning two things.

Two attributes are only available at this point. mailbox.lane records the routing decision between the dialogue consumer and the handlers reached by @AgenorMessageHandler and onDirectMessage. conversation.id is present for dialogue traffic and is what lets a whole exchange be reassembled: filter on it and you get one negotiation, in order, across every agent that took part.

Watching a conversation¶

A negotiation between agents is several messages across several agents, and reading it as a flat log is the difference between having telemetry and being able to use it. Every agent.receive span carries conversation.id, so the exchange is already grouped — you just have to ask for it.

Across nodes: filter on `conversation.id`¶

This is the one that works in a distributed deployment, and it needs no code. Each node exports its own spans; the collector is the fan-out. In Jaeger, search on the tag:

conversation.id=<the id>

You get every message of that conversation, in order, across every agent that took part and whichever node each of them runs on — a contract net's CFP, the proposals, the acceptance and the result, as one trace. mailbox.lane tells you which inbound path each message took, and agent.id which agent handled it.

The local Jaeger stack in Local development is enough to try this on a single machine with two runtimes.

Without a collector: the built-in console¶

If you are not running OpenTelemetry, the web console can show the same grouping from the message sniffer's ring buffer:

GET /api/conversations           → { "conv-1": 4, "conv-2": 2 }   ids with message counts
GET /api/conversations/{id}      → the conversation, oldest message first

Register the sniffer first — it is not started by default:

SnifferSupport.register(runtime);

and the same data is available in code:

List<StoredMessage> exchange = SnifferSupport.findByConversation(runtime, "conv-1");
Map<String, Integer> all     = SnifferSupport.getConversations(runtime);

This route is in-memory only. The sniffer captures by subscribing with a Java predicate through FilterableSubscriber, which by design no remote backend implements — a predicate cannot be evaluated server-side (ADR-020). Running against Redis, the sniffer logs "MessageDispatcher does not support predicate filtering — sniffer disabled" at startup and captures nothing. For a distributed deployment use the span route above, which goes through the mailbox and is therefore transport-independent.

Seeing what failed¶

Traces show you the messages that arrived. The ones that never did are a different question, and the console answers it on every transport:

GET /api/deadletters             → the recent entries, newest first
GET /api/deadletters?limit=20    → at most 20 of them

Each entry says what failed and why:

{
  "messageId": "3f1c...",
  "topic": "orders.created",
  "senderId": "order-service",
  "recipientId": null,
  "reason": "IllegalStateException: inventory unavailable",
  "attempts": 3,
  "deadLetteredAt": "2026-09-04T07:41:12.884Z",
  "payload": "Order[id=88123, total=42.00]"
}

The dashboard shows the same list beside the live events, with a count in the stat grid.

Unlike the conversation view above, this one is not in-memory only. It reads runtime.getDeadLetterQueue() — a port, not a Java predicate — so it answers from whatever queue the runtime was built with: the bounded in-memory buffer by default, the durable Redis DLQ stream when the runtime is given RedisMessagingFactory.deadLetterQueue(). Same page, either transport.

The same data is available in code, and the reach of each implementation is in Messaging.

Metrics reference¶

Metrics are emitted via the OTel Meter API when OTel is active. All metric names are prefixed with agenor..

Metric	Type	Labels	Description
`agenor.llm.tokens`	Counter	`provider`, `model`, `direction` (`input`\|`output`)	Total tokens consumed
`agenor.llm.requests`	Counter	`provider`, `model`, `outcome` (`success`\|`error`)	LLM call count
`agenor.llm.latency`	Histogram	`provider`, `model`	End-to-end LLM call duration (ms)
`agenor.guardrail.violations`	Counter	`guardrail_name`	Blocked guardrail evaluations
`agenor.hitl.pending`	UpDownCounter	—	Inflight human approval requests
`agenor.behavior.executions`	Counter	`behavior_type`, `outcome` (`success`\|`error`)	Behavior execution count
`agenor.directory.resolve.latency`	Histogram	—	Endpoint resolution time (ms)

Collector setup¶

Local development (Jaeger all-in-one via Docker Compose)¶

The agenor-examples module ships a ready-made agenor-examples/src/main/resources/observability/docker-compose.yml:

cd agenor-examples/src/main/resources/observability
docker compose up -d

This starts: - Jaeger — traces UI at http://localhost:16686 - Prometheus — metrics at http://localhost:9090 - OpenTelemetry Collector — OTLP/HTTP receiver on port 4318, OTLP/gRPC on 4317

Point your application at http://localhost:4318 (OTLP/HTTP) or http://localhost:4317 (OTLP/gRPC).

Running the observability example¶

# Start the collector stack first
cd agenor-examples/src/main/resources/observability && docker compose up -d

# Run the example
mvn exec:java -pl agenor-examples \
  -Dexec.mainClass="dev.agenor.examples.observability.ObservabilityExample"

Open http://localhost:16686 and search for service agenor-observability-example to see the complete trace.

Context propagation¶

OTel context propagation uses the standard io.opentelemetry.context.Context. Parent-child relationships work in two steps:

Parent makes itself current — instrumented components call span.makeCurrent() inside a try-with-resources block. This writes the span into Context.current() for the duration of the block.
Child captures the parent — OtelAgenorTelemetry.spanBuilder() reads Context.current() at call time and stores it as the parent context. Any span started from that builder is automatically linked to the active parent.

Span parent = telemetry.spanBuilder("behavior.execute").startSpan();
try (var scope = parent.makeCurrent()) {
    // spans created here (e.g. llm.chat, mcp.tool.call) are children of parent
    doWork();
    parent.setStatus(SpanStatus.OK);
} catch (Exception e) {
    parent.recordException(e).setStatus(SpanStatus.ERROR);
    throw e;
} finally {
    parent.end(); // end after scope closes — scope only removes from context
}

SpanScope.close() never throws and only pops the span from the context stack; it does not end the span. span.end() must still be called in the finally block.

Spans emitted for async operations (llm.chat, llm.chat.stream) use CompletableFuture and cannot call makeCurrent() because the async work completes on a different thread. Their parent is captured at spanBuilder() call time (step 2 above) — correct as long as the caller already has the right parent in Context.current().

Zero-cost no-op (default)¶

When OTel is absent (or agenor.telemetry.enabled: false), all instrumented components use NoopAgenorTelemetry:

spanBuilder(name) returns the same singleton builder (no allocation).
startSpan() returns the same singleton noop span (no allocation).
makeCurrent() returns the same singleton noop scope (no allocation).
All methods are no-ops that return this or the singleton immediately.

This is verified by TelemetryClasspathIsolationTest in agenor-core, which asserts that io.opentelemetry.api.OpenTelemetry is not on the core classpath.

Verifying no OTel leakage in agenor-core¶

mvn dependency:analyze -pl agenor-core

The output must not list any io.opentelemetry artifact — confirmed by the CI quality gate.