LLM Integration Guide

This guide covers the complete integration of Large Language Models into Agenor agents: provider setup, the core API, function calling, LLM-aware agents, memory management, error handling, and testing patterns.

The LLM subsystem spans three modules:

• agenor-core (dev.agenor.core.llm, dev.agenor.core.memory.llm) — provider-agnostic interfaces and records
• agenor-adapters (dev.agenor.adapters.llm) — concrete provider implementations
• agenor-runtime-llm (dev.agenor.runtime.llm.LLMAgent, dev.agenor.runtime.memory.llm) — LLM-aware base agent and memory management, split out of agenor-runtime per ADR-027

---

Package Overview

agenor-core / dev.agenor.core.llm
├── LLMProvider.java             # Core interface
├── LLMRequest.java              # Immutable request (builder)
├── LLMResponse.java             # Response with content + usage
├── LLMMessage.java              # Single conversation message
├── StreamingChunk.java          # Chunk for streaming responses
├── FunctionDefinition.java      # Declare callable functions
├── FunctionCall.java            # LLM's function call request
├── LLMMemoryAware.java          # Injection marker interface (since 0.10.0)
└── LLMException.java            # Typed error hierarchy

agenor-core / dev.agenor.core.memory.llm
├── LLMMemoryManager.java        # Interface for memory management
├── ContextWindowStrategy.java   # Strategy interface
└── TokenEstimator.java          # Token counting interface

agenor-adapters / dev.agenor.adapters.llm
├── LLMProviderFactory.java      # Factory (recommended entry point)
├── openai/OpenAIProvider.java
├── anthropic/AnthropicProvider.java
├── ollama/OllamaProvider.java
└── ToolConversionUtils.java     # FunctionDefinition → vendor schema

agenor-runtime-llm / dev.agenor.runtime.llm
└── LLMAgent.java                # LLM-powered base agent

agenor-runtime-llm / dev.agenor.runtime.memory.llm
├── DefaultLLMMemoryManager.java
├── ContextWindowStrategies.java # FIXED, SLIDING, SUMMARIZED constants
├── FixedWindowStrategy.java
├── SlidingWindowStrategy.java
├── SummarizationStrategy.java
├── ModelTokenLimits.java        # Known model context sizes
└── SimpleTokenEstimator.java

---

Available Providers

LLMProviderFactory (recommended)

LLMProviderFactory is the recommended entry point. It returns typed builders for each provider:

// OpenAI
LLMProvider openai = LLMProviderFactory.openai()
    .apiKey(System.getenv("OPENAI_API_KEY"))
    .modelName("gpt-4o")           // default: gpt-4o
    .temperature(0.7)
    .maxTokens(2000)
    .timeout(Duration.ofSeconds(60))
    .build();

// Anthropic
LLMProvider anthropic = LLMProviderFactory.anthropic()
    .apiKey(System.getenv("ANTHROPIC_API_KEY"))
    .modelName("claude-sonnet-4-6")  // default
    .temperature(0.7)
    .maxTokens(4096)
    .build();

// Ollama (no API key needed)
LLMProvider ollama = LLMProviderFactory.ollama()
    .baseUrl("http://localhost:11434")  // default
    .modelName("llama3.2")
    .temperature(0.8)
    .timeout(Duration.ofMinutes(2))
    .build();

Environment variables

Provider	Variable	Notes
OpenAI	OPENAI_API_KEY	Required
Anthropic	ANTHROPIC_API_KEY	Required
Ollama	—	Local server, no key

Direct instantiation

The builders are also accessible directly via new OpenAIProvider.Builder(), new AnthropicProvider.Builder(), or new OllamaProvider.Builder() but LLMProviderFactory is preferred.

---

LLMProvider API

public interface LLMProvider {
    // Blocking async chat — returns a single response
    CompletableFuture<LLMResponse> chat(LLMRequest request);

    // Streaming — calls chunkHandler for each arriving token
    CompletableFuture<Void> chatStream(LLMRequest request, Consumer<StreamingChunk> chunkHandler);

    // List models this provider reports as available
    CompletableFuture<List<String>> getAvailableModels();

    String getProviderName();
    boolean supportsFunctionCalling();
    boolean supportsStreaming();
}

All methods return CompletableFuture for non-blocking execution. Providers are thread-safe and can be shared.

---

Request and Response

Building LLMRequest

Since 0.16.0 the model field is optional. When omitted, the provider uses its configured model (set via modelName(...) at construction time). Supply an explicit model only when you want to route a specific request to a different model than the provider default.

// Common case — model inferred from provider configuration
LLMRequest request = LLMRequest.builder()
    .systemMessage("You are a helpful assistant.")
    .userMessage("What is the capital of France?")
    .temperature(0.7)
    .maxTokens(200)
    .build();

// Explicit per-request override — routes this request to a different model
LLMRequest expensiveRequest = LLMRequest.builder()
    .model("o3")
    .userMessage("Prove the Riemann hypothesis.")
    .build();

For multi-turn conversations, pass the full message list:

List<LLMMessage> history = new ArrayList<>();
history.add(LLMMessage.system("You are a helpful assistant."));
history.add(LLMMessage.user("What is photosynthesis?"));

LLMRequest request = LLMRequest.builder()
    .messages(history)
    .build();

LLMMessage roles

LLMMessage.system("You are an assistant.");   // SYSTEM
LLMMessage.user("Hello!");                    // USER
LLMMessage.assistant("Hi there!");            // ASSISTANT
LLMMessage.function(call, resultJson);        // FUNCTION (tool result), paired to its call

Reading LLMResponse

provider.chat(request).thenAccept(response -> {
    System.out.println(response.content());
    System.out.println("Model: " + response.model());
    System.out.println("Tokens: " + response.usage().totalTokens());
    System.out.println("Finish: " + response.finishReason());
});

Streaming with StreamingChunk

provider.chatStream(request, chunk -> {
    if (chunk.hasContent()) {
        System.out.print(chunk.content());
    }
    if (chunk.isLast()) {
        System.out.println("\n[done]");
    }
}).join();

---

Function Calling

OpenAI, Anthropic and Ollama support function/tool calling, through chat().

With Ollama, whether the model uses the functions depends on the model: those tagged tools in Ollama's library do (llama3.2, qwen2.5, ...). A request with functions to a model without tool support fails with LLMException INVALID_REQUEST, and the message says the model "does not support tools". Small models call tools less reliably than large ones. Ollama sends no id with a call, so OllamaProvider gives each call one, and the result you send back with LLMMessage.function(call, result) is paired to it.

chatStream() carries text only, so a streamed response cannot return a function call. Ollama leaves the functions off a streamed request and logs a WARN; use chat() for a tool loop.

Define a function

FunctionDefinition weather = FunctionDefinition.builder("get_weather")
    .description("Get current weather for a city")
    .stringParameter("location", "City name, e.g. Paris", true)
    .enumParameter("unit", "Temperature unit", false, "celsius", "fahrenheit")
    .build();

Send a request with functions

LLMRequest request = LLMRequest.builder()
    .userMessage("What's the weather in Paris?")
    .addFunction(weather)
    .build();

Handle function calls in the response

provider.chat(request).thenAccept(response -> {
    if (response.hasFunctionCalls()) {
        FunctionCall call = response.functionCalls().get(0);
        String location = call.getStringArgument("location");  // "Paris"
        String unit     = call.getStringArgument("unit");      // "celsius" or null

        // Execute the function
        String result = fetchWeather(location, unit);

        // Continue conversation with the function result, paired to the call it answers —
        // needed to disambiguate two parallel calls to the same function, and required
        // outright by Anthropic, which pairs tool_use to tool_result server-side.
        LLMRequest followUp = LLMRequest.builder()
            .messages(request.messages())
            .addMessage(response.toMessage())
            .addMessage(LLMMessage.function(call, result))
            .build();

        provider.chat(followUp).thenAccept(final_ ->
            System.out.println(final_.content())
        );
    } else {
        System.out.println(response.content());
    }
});

ToolConversionUtils handles the conversion from FunctionDefinition to each provider's specific JSON schema format automatically.

---

LLMAgent — Agents Based on LLM

LLMAgent (in agenor-runtime-llm) extends BaseAgent with conversation history and context window management. Extend LLMAgent instead of BaseAgent whenever your agent needs to interact with an LLM.

If the agent already extends a domain superclass and cannot extend LLMAgent, implement LLMMemoryAware (in dev.agenor.core.llm) instead. The runtime detects this interface and injects a LLMMemoryManager automatically, exactly as it does for LLMAgent:

@Agent("my-domain-agent")
public class MyDomainAgent extends DomainClass implements Agent, LLMMemoryAware {

    private LLMMemoryManager llmMemoryManager;

    public MyDomainAgent(AgentContext ctx) { ... }

    @Override
    public void setLLMMemoryManager(LLMMemoryManager manager) {
        this.llmMemoryManager = manager;
    }

    @AgenorMessageHandler("my.topic")
    public void handle(Message msg) {
        // llmMemoryManager is ready here
    }
}

LLMAgent itself implements LLMMemoryAware — no changes required for existing subclasses.

@Agent("customer-support")
public class SupportAgent extends LLMAgent {

    private final LLMProvider provider;

    public SupportAgent() {
        this.provider = LLMProviderFactory.openai()
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .modelName("gpt-4o")
            .build();
    }

    @Override
    protected void onStart() {
        // Configure context window budget
        setDefaultStrategy(ContextWindowStrategies.SLIDING);
        setDefaultConversationBudget(3000);
        setDefaultContextBudget(800);
        configureAutoSummarization(6000, 15);

        // Seed with a system message
        if (hasLLMMemory()) {
            addConversationMessage(
                LLMMessage.system("You are a customer support agent.")
            ).join();
        }
    }

    @AgenorMessageHandler("support.query")
    public void handleQuery(Message msg) {
        String question = msg.getContent(String.class);

        // Add the user turn
        addConversationMessage(LLMMessage.user(question)).join();

        // Build a prompt that fits the configured token budget
        List<LLMMessage> prompt = buildLLMPrompt(question, 2000).join();

        // Call the LLM
        String answer = provider.chat(
            LLMRequest.builder().messages(prompt).build()
        ).join().content();

        // Record the assistant turn
        addConversationMessage(LLMMessage.assistant(answer)).join();

        // Store a key fact from the conversation
        storeFact("last-query", question).join();

        getMessageDispatcher().publish(Message.builder()
            .topic("support.response")
            .content(answer)
            .build());
    }
}

Key LLMAgent methods

Method	Description
addConversationMessage(LLMMessage)	Append a message to conversation history
addConversationMessages(List<LLMMessage>)	Append multiple messages
buildLLMPrompt(query, maxTokens)	Build prompt list applying context window strategy
storeFact(key, content)	Store a fact in long-term memory
retrieveFacts(query, maxTokens)	Retrieve relevant long-term facts
clearConversationHistory()	Clear short-term history (long-term facts preserved)
getConversationMessageCount()	Current number of messages in history
getConversationTokens()	Estimated token count of the history
hasLLMMemory()	Guard: true only if LLMMemoryManager is injected
setDefaultStrategy(strategy)	Override context window strategy
setDefaultConversationBudget(tokens)	Token budget for conversation history
setDefaultContextBudget(tokens)	Token budget for retrieved facts
configureAutoSummarization(threshold, batchSize)	Auto-summarize when threshold exceeded

---

LLM Memory Management

DefaultLLMMemoryManager

DefaultLLMMemoryManager is the LLMMemoryManager implementation provided by agenor-runtime-llm. It wires together a MemoryStore (for persistence), a TokenEstimator, and an in-memory conversation list.

MemoryStore store         = new InMemoryStore();
TokenEstimator estimator  = new SimpleTokenEstimator();
LLMMemoryManager memory   = new DefaultLLMMemoryManager(store, estimator, "my-agent");

// Add messages
memory.addMessage(LLMMessage.user("Hello!")).join();
memory.addMessage(LLMMessage.assistant("Hi! How can I help?")).join();

// Retrieve with token budget and strategy
List<LLMMessage> prompt = memory.getConversationHistory(
    2000, ContextWindowStrategies.SLIDING
).join();

// Store a long-term fact
memory.remember("user-name", "Alice", Map.of("category", "profile")).join();

// Retrieve relevant context
List<MemoryEntry> facts = memory.retrieveRelevantContext("user preferences", 500).join();

Context Window Strategies

ContextWindowStrategies (in agenor-runtime-llm) exposes three pre-built strategy constants:

Constant	Algorithm	Requires LLM	Best For
ContextWindowStrategies.FIXED	Last N messages that fit in budget	No	Short conversations
ContextWindowStrategies.SLIDING	Recent + important older messages (scored)	No	Long conversations
ContextWindowStrategies.SUMMARIZED	Recent + LLM-generated summary of older messages	Yes	Very long conversations

SLIDING scoring prioritises: system messages, messages with function calls, longer messages, user messages over assistant messages.

Lookup by name at runtime:

ContextWindowStrategy strategy = ContextWindowStrategies.forName("sliding");

Custom strategy

Implement the ContextWindowStrategy interface:

public class ImportanceStrategy implements ContextWindowStrategy {
    @Override
    public List<LLMMessage> selectMessages(List<LLMMessage> all, int maxTokens, TokenEstimator est) {
        // custom selection logic
    }

    @Override
    public String getName() { return "importance"; }
}

ModelTokenLimits

ModelTokenLimits provides a registry of known model context window sizes:

int limit = ModelTokenLimits.getLimit("gpt-4o");    // 128_000
int limit = ModelTokenLimits.getLimit("claude-3-opus-20240229"); // 200_000
int limit = ModelTokenLimits.getLimit("unknown");    // DEFAULT_LIMIT (4096)

// Register a custom model
ModelTokenLimits.register("my-fine-tuned-model", 16_384);

A model not registered here does not fail — it silently gets DEFAULT_LIMIT (4,096 tokens), which can be far smaller than the model's real window and trims conversation history much earlier than necessary. SimpleTokenEstimator logs one warning per unrecognised model name the first time it is asked for that model's context window, naming the register call that fixes it; register the real limit once and the warning does not repeat.

Each provider's Models enum (OpenAIProvider.Models, AnthropicProvider.Models, OllamaProvider.Models) is deprecated as of 0.36.0 for this reason: a public enum of model constants goes stale the day the vendor ships a model it does not list, and removing a retired constant is itself a breaking API change — so maintaining one at all works against ever shipping a release with no breaking change. Name a model by its string id (LLMRequest.builder().model("gpt-5")) instead of the enum constant; every request has honoured a per-request model since 0.35.0, and each provider's getDefaultModel() still names a sensible default when none is given. getAvailableModels() is not a live query against the provider's API — it returns the fixed list of models this adapter recognises a context window for, so a model missing from it can still be requested by name; it simply will not appear in that list.

Injecting LLMMemoryManager into the runtime

Register it via AgenorRuntime.builder() so that it is automatically injected into LLMAgent instances:

var memory = new DefaultLLMMemoryManager(
    new InMemoryStore(),
    new SimpleTokenEstimator(),
    "shared-memory"
);

var runtime = AgenorRuntime.builder()
    .scanPackage("com.example.agents")
    .llmMemoryManager(memory)
    .build();

Alternatively, set it directly on a specific agent:

mySupportAgent.setLLMMemoryManager(memory);

---

Error Handling (LLMException)

LLMException carries an ErrorType a caller can branch on, and every bundled adapter — OpenAI, Anthropic and Ollama — now populates it: chat() and chatStream() both classify whatever the underlying client threw before the caller ever sees it, walking the whole cause chain rather than only the outermost frame. getStatusCode() and isRetryable() are populated too, from the same classification.

Two values cannot come from classification and stay UNKNOWN on every adapter: QUOTA_EXCEEDED has no distinct signal from RATE_LIMIT in the client libraries these adapters build on, and CONTEXT_LENGTH_EXCEEDED has no matching exception type at all — both would need guessing from a provider's message text, which is not done here. A caller that distinguishes quota exhaustion from ordinary rate limiting still has to read the message.

provider.chat(request).exceptionally(ex -> {
    Throwable cause = ex instanceof CompletionException ? ex.getCause() : ex;

    if (cause instanceof LLMException llmEx) {
        switch (llmEx.getErrorType()) {
            case RATE_LIMIT -> {
                log.warn("Rate limit hit on {}", llmEx.getProvider());
                // Schedule retry with exponential backoff
            }
            case AUTHENTICATION -> {
                log.error("Invalid API key for provider: {}", llmEx.getProvider());
            }
            case MODEL_NOT_FOUND -> {
                log.error("Model {} not found on {}", llmEx.getModel(), llmEx.getProvider());
                // Ollama: run `ollama pull <model>` first
            }
            case NETWORK -> {
                log.warn("Network error, retrying...");
                // Retry around the call
            }
            default -> log.error("LLM error [{}]: {}", llmEx.getErrorType(), llmEx.getMessage());
        }
    }
    return null;
});

Full ErrorType enum

INVALID_REQUEST, AUTHENTICATION, RATE_LIMIT, QUOTA_EXCEEDED, MODEL_NOT_FOUND, NETWORK, PARSE_ERROR, SERVER_ERROR, CONTEXT_LENGTH_EXCEEDED, CONTENT_FILTERED, UNSUPPORTED_OPERATION, UNKNOWN.

Retry pattern

isRetryable() is the retry decision already made — it comes from the same classification as ErrorType, not from re-deriving it per type: a RATE_LIMIT, a 5xx SERVER_ERROR or a NETWORK failure is retryable, an AUTHENTICATION failure or a plain 4xx is not. A daemon that is not running, such as a stopped Ollama, is NETWORK and retryable. One NETWORK case is not: a host that does not resolve, which LangChain4j reports as a configuration error.

The exception reaches you after the LangChain4j client has made its own attempts. Agenor does not set maxRetries, so the client's default applies, and they show only as WARN lines in the log. Your retry is a second layer on top: space it in seconds or minutes, not milliseconds. Wrap the provider call in a resilience library — Resilience4j and Failsafe both take a predicate over the thrown exception — and let it own the back-off:

boolean worthRetrying(Throwable t) {
    return t instanceof LLMException llmEx && llmEx.isRetryable();
}

---

Provider Configuration Reference

OpenAI builder options

Method	Default	Description
apiKey(String)	—	Required. OPENAI_API_KEY
modelName(String)	"gpt-4o"	Model identifier
baseUrl(String)	OpenAI default	Custom endpoint
temperature(Double)	0.7	0.0–2.0
maxTokens(Integer)	2000	Max output tokens
timeout(Duration)	60s	Request timeout
logRequests(boolean)	false	Log request payload
logResponses(boolean)	false	Log response payload

Name a model by its string id. The per-provider Models enums are deprecated for removal in 1.0.0; see ModelTokenLimits for why, and for what getAvailableModels() does and does not return.

Anthropic builder options

Same fields. Default model: claude-sonnet-4-6.

Name a model by its string id. The per-provider Models enums are deprecated for removal in 1.0.0; see ModelTokenLimits for why, and for what getAvailableModels() does and does not return.

Ollama builder options

Method	Default	Description
baseUrl(String)	http://localhost:11434	Ollama server URL
modelName(String)	"llama3.2"	Local model name
temperature(Double)	0.7
timeout(Duration)	5m	Longer for large local models
logRequests/logResponses	false

No API key required. Streaming is supported.

---

Testing

ScriptedLLMProvider, in agenor-runtime-llm, answers from a script instead of a model. It is held to the same contract test as the three shipped providers: it answers with the model the request names, streams, and estimates token usage. Replies come from a queue in order, then from a function of the request once the queue is empty, and every request is recorded:

ScriptedLLMProvider llm = ScriptedLLMProvider.create()
        .replyCalls(FunctionCall.of("get_weather", "{\"location\": \"London\"}"))
        .reply("It is 17°C in London.");

// ... the agent under test asks, runs get_weather, and asks again with the result ...

assertThat(llm.requests()).hasSize(2);
assertThat(llm.requests().get(1).messages()).anyMatch(LLMMessage::isFunction);

A reply can depend on what the agent sent, and a failure can be one-off or last:

llm.orElse(req -> req.messages().get(0).content().startsWith("Classify") ? "technical" : "draft");

llm.fail(LLMException.rateLimit("OpenAI", "slow down"));                           // the next call only
llm.orElse(req -> { throw LLMException.networkError("Ollama", "down", null); });   // every call from now on

With nothing queued and no orElse, a call fails and names the request that had no reply. When one provider is shared by the agents of a running system, reset() clears the queue, the recorded requests and the orElse between tests.

chatStream() takes from the same queue and delivers the text word by word. A queued function call fails the stream instead, because a StreamingChunk carries text only.

Mockito or this. For a unit test of one class that calls chat() once, a Mockito mock is fine. ScriptedLLMProvider is for the case Mockito writes badly: a real agent, on a real transport, whose next request depends on what the last reply made it do. The tool loop above is that case.

---

See Also

• Agent Development Guide — LLMAgent lifecycle and behaviors
• Architecture Guide — module overview and concurrency model
• Dialogue Protocol — A2A and structured agent communication
• agenor-examples/README.md — runnable LLM examples