bestmultiagent

Published 2026-11-18

Glossary for judging multi-agent frameworks

Annotation queue

An annotation queue is a work list of agent runs that people review and label so their judgment becomes reusable feedback. You meet one when automated scores cannot judge subjective quality. LangSmith says annotation queues enable structured collection of human feedback and let you flag specific runs for review. You can then transfer annotated runs to datasets for future evaluations.

Background run

Background run is an agent execution that starts and keeps going without the caller waiting on the response, which suits long jobs. The caller usually checks its status later. LangChain's docs say kicking off background runs can be useful for long running jobs. Agno lets you pass background=True to Workflow.arun() to start a detached asynchronous task.

Batch API

Batch API is an interface that accepts many model requests together and returns the results later instead of answering each one immediately. Engineers use it for bulk evaluations and other work where nobody waits on the reply. Anthropic's Message Batches API offers a 50 percent discount on all usage compared to standard API prices, and batch results are available for 29 days after creation.

Breakpoint

A breakpoint is a fixed point in a graph where a run pauses so a developer can inspect state before going on. Engineers meet them mostly in debugging and testing, for example when stepping through a LangGraph build. In LangGraph, you can use static interrupts as breakpoints to step through graph execution one node at a time. Its docs say static interrupts are triggered at defined points either before or after a node executes.

Context compaction

Context compaction is replacing older parts of a long conversation with a summary so the work can continue inside the model's limit. Agent builders meet it in long-running tasks that approach that limit. Anthropic's docs say compaction automatically summarizes earlier parts of the conversation on the server so the conversation can continue past the limit, and that it is available in beta for Claude 4.6 and later models and Claude Mythos Preview. Test whether the summary keeps the details your task depends on.

Context engineering

Context engineering is choosing which instructions, tools, data and history go into a model's context on each call. Agent builders meet it once a run loops over many turns, and it falls under context handling in our criteria. Anthropic's engineering blog defines it as the set of strategies for curating and maintaining the optimal set of tokens during LLM inference, including information outside the prompts. It adds that context engineering is iterative, with curation happening each time you decide what to pass to the model.

Context window

Context window means the amount of text a model can work with in one request, counting both what you send and what it writes back. Engineers hit it in long agent loops. Anthropic's docs define it as all the text a language model can reference when generating a response, including the response itself, and give a capacity of up to 1M tokens, depending on the model. The same page says everything in the request counts toward it, including the system prompt, messages, tool results and tool definitions.

Cron job

Cron job is a task set to run on a fixed schedule, such as every hour or every morning, with no person starting it. For agents it covers recurring work like a nightly data sync or a daily summary report. When you compare frameworks, check where the schedule is stored and whether a failed scheduled run is retried or reported to someone who can act on it.

Durable execution

Durable execution is the ability of a workflow to survive crashes and restarts and continue from recorded progress instead of starting over. Engineers meet it when an agent run lasts long enough that a lost process would otherwise lose the work. Temporal says a Workflow Execution is durable because it runs your application code effectively once and to completion, whether that takes seconds or years. In Pydantic AI's durable execution docs, each engine journals the model requests and tool calls it wraps as its own durable units.

Evaluation dataset

An evaluation dataset is a fixed set of test cases you run an agent against to measure quality before release. Start with hand-written examples, then add cases taken from real traffic. LangSmith defines a dataset as a collection of examples used for evaluating an application, where each example is a test input and reference output pair. Its offline evaluations target examples from datasets, which it calls curated test cases with reference outputs.

Event history

Event history is the ordered record of what a workflow has already done, kept so the run can be rebuilt after a pause or failure. Buyers meet it when asking how a platform recovers long-running work. Temporal keeps an Event History that its docs describe as a complete, ordered log of everything that has already happened in a Workflow. To resume, Temporal starts the Workflow code from the beginning and replays the Event History step by step to guide the code back to its earlier state.

Fallback model

Fallback model is a second model a system switches to when the first one fails or cannot answer. It matters for reliability, because a provider error should not end a user's session. Microsoft Foundry's model router handles this itself: if the selected model returns a retryable error, it fails over to the next-best model automatically, with nothing to configure.

Feedback

Feedback is a judgment attached to an agent's output, from a user or a reviewer, saying whether the result was good. Engineers use it to find weak runs and to build evaluation sets from real traffic, which is part of observability for agent systems. LangSmith's observability docs describe how to annotate outputs and gather user feedback using queues or inline annotation.

GenAI semantic conventions

GenAI semantic conventions are shared attribute and metric names for telemetry about generative AI calls, so model and agent work is labeled the same way across tools. OpenTelemetry describes semantic conventions as a standard naming scheme for common telemetry data types, and its docs say the GenAI conventions have moved to a separate repository. Agent Framework's Python setup installs opentelemetry-semantic-conventions-ai by default and creates a metric named gen_ai.client.token.usage that measures token usage.

Groundedness

Groundedness is the degree to which an agent's answer is supported by the context it was given, such as retrieved documents or tool results. Low groundedness points to hallucination, which hurts most in agents that answer from retrieval. Check it alongside answer quality, since a fluent answer can still rest on nothing the agent was given. ADK includes a criterion, hallucinations_v1, described as LLM-judged groundedness of agent response against context.

Interrupt

An interrupt is a point where an agent run stops and waits for outside input, usually a person's approval, before continuing. Teams use it for approval steps, which our methodology treats under human-in-the-loop. In LangGraph, interrupts pause graph execution at specific points and wait for external input, and they work by calling the interrupt() function in graph nodes. LangGraph's interrupts page lists a rule that side effects called before interrupt must be idempotent.

LLM-as-a-judge

LLM-as-a-judge is an evaluation method in which a language model scores another model's output against a prompt or rubric. Teams use it when exact-match checks cannot assess free-form answers. LangSmith says LLM-as-a-judge evaluators use an LLM to evaluate traces as a scalable substitute for human-like judgment. ADK lists criteria that rely on a model judge, including an LLM-judged semantic match against a reference response.

Model routing

Model routing is the practice of sending each prompt to whichever model suits it, rather than using one model for every request. Buyers meet it as a cost lever, since easy prompts rarely need the largest model. Microsoft Foundry's model router is a routing model that judges how demanding each request is and sends it to the model best suited to answer it.

Online evaluation

Online evaluation scores live production traffic instead of a fixed test set, so you catch quality drift after release. Each judge call adds model cost, so sampling deserves a decision up front. LangSmith says online evaluations provide real-time feedback on production traces and that they target runs and threads without reference outputs. You can also configure a sampling rate for a LangSmith online evaluator.

OpenTelemetry

OpenTelemetry is an observability framework and toolkit for generating, exporting and collecting telemetry such as traces, metrics and logs. It is open source and vendor-agnostic, so it can be used with a broad variety of backends, though it is not an observability backend itself. It is a CNCF project formed by a merger of OpenTracing and OpenCensus. In ADK for Kotlin, you can enable observability by configuring OpenTelemetry for traces. Our guide to agent observability applies these terms to framework choice.

Pending writes

Pending writes are partial results saved from steps that finished before a sibling step failed, so recovery does not repeat them. They matter when parallel steps call paid models or tools. LangGraph says that when a node fails mid-execution at a super-step, it stores pending checkpoint writes from other nodes that completed successfully there, so resuming from that super-step does not re-run the successful nodes.

Prompt caching

Prompt caching is reusing the processed form of an unchanged prompt prefix across requests to cut input cost and latency. Buyers meet it as a line in model pricing and as a constraint on how prompts are ordered. OpenAI's docs say it reuses work when requests share the same prompt prefix, and that pricing varies by model.

Prompt management

Prompt management is the practice of storing prompts outside application code, with version history, so they can be changed and reviewed on their own. Buyers care because a prompt edit then does not need a new release. LangChain's deployment docs describe how to update prompts and contexts without redeploying, by managing the prompts and versioned contexts that deployed agents pull at runtime.

Rate limit

A rate limit caps how many requests or tokens a client can send to a service in a set time window. Agent teams hit one when several agents call the same model provider at once. Kong's AI Rate Limiting Advanced plugin is only available as part of its AI Gateway Enterprise offering. It supports three strategies to calculate the number of tokens. Its Retry-After headers will be present on 429 errors to show how long the service is expected to be unavailable.

Recursion limit

Recursion limit is a cap on how many steps a looping agent graph may take, so a loop that never meets its exit condition cannot run forever. Engineers meet it when an agent keeps cycling between a model call and a tool call because the exit check never passes. LangGraph's build checklist says a loop can cycle until a termination condition, or cap runs with a recursion limit.

Retrieval-augmented generation

Retrieval-augmented generation, or RAG, is a pattern where a system fetches relevant documents and passes them to the model so its answer rests on that material. Teams use it to ground answers in their own content. LangChain's agentic RAG tutorial calls its answer node the classic RAG step, which combines the user question with the retrieved context and asks the model for a grounded reply. In that tutorial, retrieval runs only when the model requests it.

Self-hosting

Self-hosting means running a tool on infrastructure you control instead of using the vendor's hosted service. It comes up when data residency, air-gapped networks or internal policy rule out a shared cloud. n8n says you can self-host n8n on your own infrastructure, on-premises, or in a private cloud. LangChain documents a standalone option to deploy Agent Server with Docker, Compose, or Kubernetes, bringing your own PostgreSQL, Redis, and LangSmith license.

Span

A span is one timed unit of work inside a trace, such as a model call or a tool call. A span can point to a parent span, which gives a trace its tree shape. OpenTelemetry says child spans represent sub-operations when spans are nested. Mastra represents each operation as a span and groups related spans into a trace, so a developer can inspect the full execution path of one request.

Streaming

Streaming means sending output or progress to the caller as it is produced, instead of after the whole run finishes. Long agent tasks and chat interfaces depend on it, because users wait on partial results. LangGraph lists streaming among its orchestration capabilities, next to durable execution and human-in-the-loop. Between agents, A2A supports real-time communication using Server-Sent Events for tasks that produce incremental results or ongoing status updates.

Suspend and resume

Suspend and resume is pausing a workflow step, saving its state, and continuing later from the same point. Teams comparing Mastra with graph frameworks meet it in approval flows and callbacks. Mastra's docs say you can pause a workflow at any step to collect more data, wait for an API callback, throttle a costly operation, or request human input. They also say suspension saves the current execution state as a snapshot, which resume restores from a specific step ID.

Task queue

Task queue is a waiting line of work items that separate worker processes pull from, so the sender does not call a worker directly. It is the piece that lets workers scale out and lets queued work outlast a worker crash.

Time to first token

Time to first token is the delay between sending a request to a model and receiving the first piece of its output. It matters most in chat and streaming agents, where a user sees a blank screen until text starts. OpenAI's latency guide defines TTFT as how long an API request takes to start returning content. The same guide treats end-to-end latency as a separate measure of how long the user waits for the complete result.

Time travel

Time travel is replaying or branching from an earlier saved point in an agent run to inspect or change what happened afterward. Teams use it to debug a bad step without rerunning the whole job. LangGraph's checkpointer docs say checkpointers let users replay prior graph executions to review or debug specific graph steps. They also make it possible to fork the graph state at arbitrary checkpoints to explore alternative trajectories.

Token usage

Token usage is the count of input and output tokens a request consumed, which drives cost and shows how much room is left for later turns. Teams read it from API responses to track spend. Anthropic's docs say every response reports what the request consumed in its usage field, and that with prompt caching the input count is split across input_tokens, cache_read_input_tokens and cache_creation_input_tokens. To estimate a request before sending it, they point to the token counting API.

Trace

A trace is the record of one end-to-end operation, such as a single agent run, built from smaller timed steps. You read traces to find which model call or tool call went wrong. The OpenAI Agents SDK says traces represent a single end-to-end operation of a workflow and are composed of spans. In that SDK, tracing is enabled by default. Tracing is unavailable for organizations that use OpenAI's APIs under a Zero Data Retention policy.

Trajectory evaluation

Trajectory evaluation scores the steps an agent took, such as which tools it called, and not only its final answer. It matters in multi-agent builds, where a correct answer can hide a wasteful or unsafe path. ADK describes the trajectory as the sequence of steps taken to reach the solution. Its tool_trajectory_avg_score criterion is an exact match of tool call trajectory.

Which sources does this page cite?