Re-Architecting the Agent Loop for Sub-Second Latency in Conversational AI

To meet a strict sub-second response threshold for an interactive tutoring system, engineers replaced standard synchronous tool loops with a decoupled streaming execution harness. The architecture achieves real-time performance through asynchronous dual-agent orchestration, speculative branch pre-generation, and parallelized safety classification.

Decoupling Generation from Execution

Standard LLM agent architectures rely on a synchronous tool loop where the model outputs a tool call, waits for execution, observes the result, and determines the next step. In production, this pattern introduces unacceptable latency. Frontier models exhibit a time-to-first-token latency of 2 to 3 seconds, followed by decoding speeds of roughly 30 tokens per second. Combined with network round-trips and audio playback, a sequential tool loop causes 3 to 4 seconds of idle time between conversational turns.

To bypass this bottleneck, the system utilizes a custom harness that decouples model generation from action execution. Instead of executing serial tool calls, the model streams multiple actions within a single response. A dedicated interpreter parses and executes each action dynamically while the downstream actions are still being generated by the model. This pipelined approach reduces the user-facing latency to the generation of the first action—approximately 30 tokens—rather than requiring the system to wait for the complete response sequence to generate.

This decoupling also allows the runtime environment to dynamically restrict or permit specific actions based on the application state. For example, during active question phases, the system filters the model's available options to prioritize scaffolding prompts over direct answers. Furthermore, the architecture processes action validation in a non-blocking manner; the execution path only halts or triggers a re-generation if the parser encounters an invalid action in the incoming stream.

Dual-Agent Concurrency and Shared State

To preserve instruction-following quality without sacrificing response speed, the architecture divides tasks between two distinct agents: a converser and a planner. The converser manages the real-time interaction with the child and operates within a highly restricted action space to prevent instructional errors, such as prematurely revealing answers. The planner runs asynchronously, evaluating the conversation history against pedagogical goals to continually adjust the converser's context window.

Running these agents concurrently introduces shared-state coordination challenges. To prevent race conditions and eliminate the overhead of distributed locking, the system records all interactions—including verbal turns, screen taps, and user interface updates—as immutable events in an append-only transaction log. Both the converser and the planner read from and append to this log independently without blocking or explicit synchronization. The planner runs on a larger, more resource-intensive model, leveraging the time the user spends thinking or speaking to run expensive reasoning cycles asynchronously.

Speculative Generation for Closed-Ended States

When the converser initiates a closed-ended interaction, such as an equation or a fill-in-the-blank prompt, the future state space is highly predictable. The harness exploits this predictability by generating speculative response branches before the user provides input.

During the idle period while the user formulates an answer, the harness hypothesizes the most probable responses. For each predicted answer, it forks the primary trajectory and pre-generates a corresponding response branch. When the user eventually responds, the system performs a low-latency match against the pre-generated branches. If a match is found, the system plays back the pre-generated response immediately, reducing the response latency to zero model-generation cycles. The trade-offs of this approach are increased API inference costs and occasional mispredictions, which require the system to fall back to real-time generation.

Parallel Guardrails and Eager Execution

Standard safety architectures process moderation filters in series with model inference, which typically adds 500 to 1000 milliseconds of latency. To hide this overhead, the system parallelizes safety classification with response generation.

As soon as user input is received, the harness concurrently dispatches the safety classifier and a low-latency model. This small model immediately generates a rapid, conversational eager response, such as mirroring or acknowledging the user's input. While this eager response is being generated and played back, the slower safety classifier completes its analysis.

Under normal operating parameters, the classifier returns a safe status before the eager response finishes, unblocking the execution of the main converser stream. However, if the classifier flags the user input as unsafe, the system immediately aborts the eager response. The system then discards the generated converser sequence and injects alternative, safety-compliant generation instructions to handle the exception gracefully.