Larger Than Language

01Why these notes exist

The models can talk. That problem is, in the narrow sense, solved: several speech-native systems now hold a fluent real-time conversation, and Knowing When to Speak is our own account of what it takes to make one feel present. What has not happened is the second half. Talking to an application is still almost entirely talking about it. The voice describes; the interface waits. You ask for a comparison and receive a paragraph. You ask it to open a page and it reads you a summary of a page you cannot see. You ask it to close the thing on your screen and it has no idea what is on your screen.

We call the missing half Interactive Intelligence: intelligence you can watch think. Not a chat window that renders markdown, and not an agent that goes away and comes back with a result — a surface that assembles itself during the sentence that explains it, and that the speaker can then act upon by name, because the surface has told the model what it contains.

The interesting question is not whether a model can call a tool. It is what an application must publish about itself before a voice can act inside it.

The gap is measurable, and someone else measured it. τ-Voice, published in March 2026, put the same grounded tasks to the same models over text and over a full-duplex voice channel: 85% pass@1 on text against 31–51% in clean voice, and 26–38% in noise — with 79–90% of the failures attributable to agent behavior rather than transcription^[17]. The voice channel is not a thin skin over a competent text agent. Something structural is lost in the translation, and we think the thing that is lost is precisely what does not survive being routed through language.

Three surfaces are described here — Live UI (§08), Speech-to-Action (§09), and Lens (§10) — but they arrive late in these notes on purpose. §02–§07 are the argument the surfaces are evidence for: what a model class is, what an Actra predicts, what an Octra predicts, and why an interface had to exist before either could be trained against anything.

02What a model class actually is

“Foundation model” describes the role and breadth of a model, not the fact that it is a language model. A foundation model is one trained over data broad enough to serve as a reusable base for many downstream capabilities, and the substrate underneath can be language, speech, vision, video, sensory data, trajectories, control, code, spatial state, tool interactions, agent behavior, or any combination. Foundation Model ≠ Large Language Model. An LLM is one species.

“Large” got attached to “language” by history rather than necessity — scaling transformers over text was what worked first, and the adjective came along for the ride. There is nothing preventing large speech models, large action models, large behavior models, large world models or large orchestration models, and the useful question was never whether a system contains text. It is: what is this model's native representational space, training objective and output space?

Two of those rows have names at Labs, because you cannot measure a class you cannot refer to.

2.1  Why Transducer, and not Transformer

A transducer is named for the conversion it performs, not for the topology that performs it. An Actra converts speech and world state into action; an Octra converts an objective and a set of resources into a policy. Neither name commits us to a decoder stack, which matters because §06 argues these families should be free to be state-space models, diffusion decision models, reinforcement-learning policies or neuro-symbolic hybrids — and a family whose own name says Transformer is arguing with itself.

The lineage is real and we would rather cite it than be discovered by it. Transformer sequence modeling^[16] produced language models; language models plus embodied action produced the vision-language-action class, where RT-2, OpenVLA and π₀ all make the same move — emit actions as tokens in the same stream as perception, so the policy is learned rather than orchestrated^[13,14,15]. Actras are to speech what VLA models are to vision. And in the terms that made the last decade legible: what GPT — the generative pre-trained transformer — named for generative language, GAT — the Generative Action Transducer — names for generative action, and GOT — the Generative Orchestration Transducer — names for generated coordination. Actras are the Labs GAT family; Octras the Labs GOT family. Attention is an excellent mechanism for all of this and we expect to keep using it. It is ancestry, not a constraint.

03What an Actra predicts

The weak version of speech-to-action is a pipeline: pipelinespeech → transcript → language model → tool call → action. Four lossy translations, three of which exist only because the middle component thinks in words. The transcript discards prosody, hesitation and emphasis. The language model receives no environment state unless someone remembered to serialise it into the prompt. The tool call is a string the model has no way to verify. And the action is executed by something else entirely, which reports back — if at all — in another string.

The strong version collapses the middle: nativespeech + context + environment → action trajectory. Linguistic structure is still understood; it simply is not required to surface as a transcript before anything can happen, any more than a VLA model renders a caption before it moves an arm.

An action-native model would optimise for a different list than next-token likelihood: intent identification, action selection, parameter grounding, action sequencing, temporal reasoning, state-transition prediction, interruption handling, multimodal grounding, permission boundaries, execution confidence, recovery behavior, and delegation. Several of those have no natural expression as text at all. Execution confidence is a scalar the model should hold about its own act; today it is a sentence we ask a language model to write about itself, which is a different and much weaker thing.

3.1  An action vocabulary

If actions are the prediction target they need a vocabulary, in the same sense that words are a vocabulary — a discrete space over which a distribution can be placed, sequences composed, and errors measured. Ours is small and application-shaped because it grew out of one application; a foundation-scale version would be neither.

Open problem — where the trajectories come from. A mature Actra would learn from human-computer interaction trajectories, voice-action pairs, application navigation, API traces, agent runs, successful and failed executions, robotics data, computer-use environments, simulated worlds, user corrections and workflow histories. Almost none of that exists as a curated public corpus, and the failure half — which is where recovery behavior has to come from — is systematically the half nobody records. We have begun recording our own: the surface now keeps a trajectory ledger — every tool invocation, its timing and whether it landed, success and failure both, session by session. It is nowhere near enough, but it is the right shape, and it records the failure half on purpose.

04What an Octra predicts

The orchestration question is not “which model is best at coding.” It is: given this objective, this context, these available models, agents, tools, compute and costs, and these latency and budget constraints — what should execute what, when, where, and in what sequence? The output is not a token. It is a policy.

Run from live speech — the visitor's spoken intent steering those decisions directly, at conversational tempo — this class is Speech‑to‑Orchestration (STO): the orchestration twin of Speech‑to‑Action, and the form in which this system runs it. STA decides what the surface does; STO decides which intelligence does it. Throughout the platform, both arrive spoken: Speech‑to‑Action Voice Commands — the unit of spoken work an STA Model interprets and executes — and Speech‑to‑Orchestration Voice Commands, the unit of spoken coordination an STO Model runs as policy.

4.1  Routing is software; orchestration is learned

A router is a ladder: if coding, the coding model; if image, the image model; if research, the research model. It is written by a person, it is correct exactly as far as that person anticipated, and it cannot get better from use. Every system that calls itself an orchestration layer today, ours included, is somewhere on that ladder.

What a policy becomes when it runs deserves its own name, because it is a different kind of object from the model that produced it. Orch Models are intelligence; AI Orchs are execution structures. An AI Orch — an AI Orchestrator, interchangeably an AI Orchestration — is the executable coordination object an orchestration policy instantiates as: it binds the participants — models, agents, tools, workflows, memory and context, data, permissions, compute, runtime environments — together with their dependencies, state and execution logic into one live operation, and it can be created, modified, paused, resumed, forked, observed and completed. The relation is the one every runtime distinction in computing has needed: an LLM can generate the reasoning for a conversation, but the conversation is a separate object with messages, state and participants — and in the same way, an Orch Model produces and governs an AI Orch, and the AI Orch coordinates the agents, models and tools. Neither requires the other: an AI Orch built from rules, dependency graphs and policy — which is exactly what this system's retrieval pipeline, fallback ladder, shared-browser sessions and media roster are — runs deterministically today, and an Octra is what makes it adaptive: the model that generates, reroutes and repairs AI Orchs instead of following the written ones. The hierarchy beneath this paragraph is the note's shortest summary: models reason, agents act, Orchs coordinate — a model thinks, an agent acts, a workflow follows predefined steps, and an AI Orch dynamically coordinates all of the above, state-aware, policy-aware and environment-aware rather than a static automation graph.

Open problem — the benchmark that would settle it. An Octra's claim is that a learned policy beats a hand-written one. Nobody can currently evaluate that, because no benchmark scores routing decisions under a real-time constraint with a human holding the floor. We have started one (§16), and we are aware that starting your own benchmark is the oldest way in the field to win an argument. Ours publishes its tasks, its rubric and its recordings for exactly that reason.

05Three centres

This is the shortest section, and the one the title is drawn from.

Put plainly: the field has spent several years asking how to make a better language model, and the answers have been extraordinary. The question we find more interesting is the one underneath it — what happens when language is no longer the primitive around which the architecture is organised? We do not know. We know that when we built a system whose job was to act on a live surface while speaking, every hard problem we hit was one that language was in the way of.

06A family is not a topology

Naming a model family after an architecture is a bet that the architecture will outlive the idea. It usually does not. Actra and Octra name an architectural philosophy — what the model is organised around — and deliberately not a network shape.

Breadth is the second axis, and it is what separates a capable model from a foundational one. A narrow Actra is trained against one application's affordances and is useful in that application. A Foundation Actra is pretrained broadly enough to act in surfaces it has never seen. The same distinction holds on the other side: an Octra that routes among four known models, against a Foundation Octra that orchestrates models, agents, tools and environments it was not written for.

07Why the interface had to be built first

There is an obvious objection to everything above: if the argument is about models, why is the rest of these notes about an interface?

The interface conviction predates the model argument. The founding interface thesis of this research record is Maya Davis’s Interactive Perspective Spheres — the sphere of integrated generative capability a person steps into, prompts orchestrated rather than typed, driven first by speech — written before the surface below existed, and the reason it was built. What the sphere named as instruments, Speech‑to‑Action now carries as spoken acts; what it named as coordination, Speech‑to‑Orchestration now carries as the floor discipline these notes measure. The era it opens — the Interconnected Generative Era — is the horizon the LAIMA program builds toward.

Because an action-native model cannot be trained against nothing. It needs a surface that declares what can be done to it, an action vocabulary over that surface, and a signal of whether the action landed — and none of the three existed. There is no accessibility tree for capabilities. There is no standard by which an application tells a model what it affords. There is no convention for reporting that an action was attempted but not verified. So we built one application that has all three, badly and by hand, because that was the only way to find out what they need to contain.

We are describing the specification by having implemented it the hard way — which is the only way anyone ever gets one.

What follows is that implementation: the surface (§08), the action layer over it (§09), the shared browser (§10), and the orchestration underneath (§11). Read them as a requirements document written in working code.

08The interactive surface

Live UI is a panel beside the conversation in which artifacts — charts, timelines, comparisons, tables, small purpose-built interfaces — are created and mutated by the model as it speaks. The visitor does not ask for a visual. Asking for one is a failure mode: it means the surface was too slow to be part of the answer.

Artifact ALive UI, unretouched

One stage, one answer: a focal chart, a supporting timeline, and a metric row — assembled while the voice was still explaining them. Screen capture from the running system.

8.1  The atomic unit is not the artifact

The first design we tried treated a visual as a result: the model finishes thinking, the panel renders, the voice narrates it. It reads as a slideshow — the visual arrives, then a description of the visual arrives, and the two are obviously separate media stapled together. What actually feels like one act is a triple: one visual beat, one narration beat, and one reveal moment that binds them. The reveal is the primitive, not the render.

So a created artifact carries a hint about when it should appear: immediately, with the narration, or after it. The panel holds the card until the speaking edge the hint names — a rising edge for with, a falling edge for after — and only then commits it. Because a speaking edge is inferred and can be missed, every hold carries a maximum: bound4 s for with narration, 12 s for after, plus a flush at session end. The conductor fails open. A lost edge costs a slightly early reveal; it never costs a card.

8.2  Hierarchy is computed, not declared

A model asked to mark its own focal artifact will mark nearly all of them focal. So the panel scores every artifact independently — a dense metric group outranks a lone paragraph — and enforces a viewport budget on every arrival: boundone hero, at most two expanded, everything else folded. If a support artifact outscores the declared focal by a margin, it is promoted over the model's stated intent; if the hero is dismissed, the next-strongest is promoted in its place. The stage repairs its own hierarchy.

A folded card is not silently hidden. Folding emits an event, and the loop converts it into a note the voice can act on, so that what the viewport withheld is carried in speech instead. This is the small rule that keeps the two media in agreement: the panel may withhold, but the conversation may not.

8.3  Two generation lanes, one truth gate

Artifacts arrive from two places. The lead lane is the conversational model's own action calls. The second is an ambient composer: a cheap, non-live text model fed the sentence boundaries of both voices, which may emit at most one small operation per beat — create, patch, or nothing. It exists because the lead lane is busy speaking, and action compliance mid-monologue is the known soft spot; the composer is the backstop that keeps the surface moving. It is bounded hard: bounda six-second minimum gap, never concurrent, a per-session call ceiling, and its own id namespace so it can only ever patch its own artifacts.

Both lanes pass the same validator, and the validator's job is to refuse rather than to render. A chart with one data point is refused with a redirect to a metric, because a single bar is a decoration pretending to be evidence. An empty table is refused, because an empty table is not a table. Labels must match values. Imagery must be https and must not be a placeholder. Any figure the model did not retrieve must be declared as an estimate, and the renderer prints an ESTIMATED chip beside it — visual truth versus visual plausibility, decided in the component rather than in the prose.

One test in the suite deserves mention because it prevents an entire failure class rather than a bug: the vocabulary-parity check extracts the component list from the instruction the model is given and asserts that each named type has a renderer on the client. When the two drift, the model is offered a type that silently fails to draw, and it looks unreliable while being perfectly obedient. The parity test makes that state unreachable.

Open problem — a grammar for visual candor. Our refusals are a written list: two points minimum, no empty containers, no non-https imagery, estimates must be labelled. That list encodes a small part of a much larger idea — that a visual makes claims which can be checked against the evidence available to the speaker. We would like a general account of that: given a claim, an evidence set and a component vocabulary, decide which renderings are supported, which are merely plausible, and which are misleading. Chart-selection heuristics answer the aesthetic question; nobody has answered the epistemic one.

09Speech-to-Action, concretely

This is the section §03 exists for. Once a surface builds itself, the visitor immediately tries to talk to it — close that, pause it, go back to the first one — and every one of those utterances is outside the reach of a conventional tool schema, because the referent is a thing that exists only on screen and only right now.

9.1  The surface publishes its own affordances

The obvious implementation gives the model a DOM. It is wrong in three independent ways: it is enormous, it is a live injection surface, and it invites the model to emit selectors — strings that are unverifiable, unstable, and wrong in ways that look right. We inverted it. Every card, on mount, registers what it can do in a small registry scoped to its own lifetime; unmounting sweeps the namespace, so a verb is addressable exactly while the thing it acts on is on screen. The registry is published to the model as a short numbered catalog of labels and aliases — never markup — and the model acts by number or name. It never names an element, because it has never seen one.

The consequence is worth stating plainly: stale actions cannot exist. The most common failure in voice control of an interface is a command that was valid a moment ago, and this design makes that state unrepresentable rather than defended against.

Artifact BA published affordance

A track retrieved in audio-only mode. The card carries a Show the video control, and by carrying it, registers it — the same verb is now addressable by speech, by that name, for exactly as long as this card is mounted. The panel header reports the media state the model is permitted to describe.

9.2  Reference resolution is a state machine, not a match

“The second one.” “No, the other one.” “Actually, cancel that.” Each of these is a move in a small dialogue game with memory, well beyond a keyword lookup. Resolution runs longest-alias-first over the catalog, and when a reference genuinely matches several candidates the system is required to ask rather than guess — ambiguity is a state, not a tie-break. Ordinals resolve against the last set offered. Corrections are checked before ordinals, which sounds like an implementation detail and is not: “no, the other one” contains the word one, and an ordinal matcher that runs first will confidently select item #1 while the visitor is telling it that it chose wrong.

Anything destructive is held for confirmation, and the hold expires. boundA confirmation is live for ninety seconds and dies after two unrelated utterances or one spoken decline. This exists because of the shape of voice: several turns complete inside a monologue, and a “yes” that arrives four exchanges later is answering a question the visitor has forgotten. A stale yes must not be able to clear a workspace. We also had to write the confirmation instruction in unusually blunt language after observing a held action announced as done — the hold is now described as nothing has happened yet, explicitly.

9.3  Verified means seen

Every registered action carries an optional verifier: a predicate evaluated against the real DOM boundafter a settle delay, once the animation the action triggered has finished. The outcome is one of verified, attempted, blocked or failed, and success may be claimed on nothing but verified. Where no verifier is possible — a fifteen-second seek has no stable observable — the accurate answer is attempted, and the voice says so.

Artifact CVerification, on camera

The visitor says pause it. The voice answers “Pausing the video. Done, paused.” — and it is allowed to say done only because the panel re-read its own state and the header changed. Two frames, six seconds apart, from one unedited session recording. This is the entire verification contract in one image: the report is downstream of the observation, not of the intent.

Why this is not over-engineering. The work found a real instance of exactly the failure it prevents. A legacy phrase-matching path was reporting success for “get rid of that chart” while removing only cards of an older type — the visitor watched the chart stay on screen while being told it was gone. A system that acts on your interface must never be the last authority on whether it acted, and a repeat failure has to stop rather than loop: boundat two consecutive failures on the same target, stop and say what happened.

Artifact DModality switched by voice

After show the video: the same track, now carrying both a video surface and the original audio card. The affordance the visitor named in Artifact B was registered by the card that owned it — and stopped existing when it did.

Open problem — the affordance interface. There is no standard by which an application declares what it can do to a model — no equivalent of an accessibility tree for capabilities rather than elements. We invented one that suits our panel; it does not generalise past our own code. The research question is what that interface should be, such that a model trained against it transfers to applications it has never seen, and such that publication is safe: the catalog is a description of power, and any surface that publishes power must reason about who is allowed to invoke it.

10Lens — a browser as a conversational object

The third surface is the one visitors ask for first: open this site and let's look at it together. Lens is a headless browser driven from the loop whose output is delivered to two consumers with deliberately different diets.

Artifact ELens, unretouched

A real page, live, inside the conversation — with back, scroll and end controls the visitor can use directly, and a voice that is looking at the same pixels.

10.1  Pixels for the human, a digest for the model

The visitor receives a screenshot, because a person wants to see the page. The model receives a bounded structural digest — boundheadings, navigation labels, buttons and main text, each capped — and the same screenshot, because reading a page and seeing its layout are different competences and we need both. Perception is change-gated: the image is hashed, and only a changed page enters context. There is no polling loop. This single decision is the difference between a browser that costs a few frames per interaction and one that floods the conversation with near-identical images until the context is exhausted.

The literature here is instructive and we followed it: agents driving browsers do markedly better acting over structured text than over pixels, which is why the accessibility-tree approach has become standard for tool use^[11,12]. Our split takes both sides of that result rather than choosing — structure to act, pixels to discuss — because the second consumer here is a human being, and no digest is a substitute for seeing what a site actually looks like.

10.2  Absence as a security property

Lens cannot type. There is no typing endpoint anywhere in its API — not a disabled one, not a permissioned one. Sign-ins, forms, checkout and anything resembling data entry are therefore not declined by policy; they are unreachable. We think this is the correct default for the first generation of browsing agents and we would rather be criticised for a narrow capability than defend a broad one: a policy that refuses is a probability, and an endpoint that does not exist is a proof.

Two further properties are structural rather than behavioral. Each session runs in its own ephemeral browser context, disposed at close, so cookies and storage from one visitor cannot be observed by the next — the browser process stays warm, the identity does not. And the address checks run twice: once before navigation, against the resolved address, and again after the page loads, against where the browser actually ended up. The second check is the one that matters, because a public URL is free to redirect somewhere private, and a pre-flight check alone would have been theatre.

Everything the page returns crosses the same boundary as any retrieved content: it is data, not instruction. The external-content envelope from Knowing When to Speak applies unchanged, and the page digest is wrapped in it. A page that addresses the assistant directly is content to describe, not a voice to obey.

Open problem — consent for a surface that can act. Read-only is a boundary we chose, not one we derived, and it is temporary: the useful version of this capability eventually books, submits and signs. We do not have a satisfying model of consent for that. Per-action confirmation does not survive contact with a real conversation — it converts a fluid exchange into a permissions dialog — and blanket authorisation is exactly the thing that should never exist. What we want is a calculus in which the human's authority is scoped, revocable and legible mid-sentence, and we do not have it.

11Orchestration under a floor constraint

Underneath all three surfaces sits the Octra layer. Its problem is not routing in the abstract. It is routing while a human is listening, which changes the character of every decision because latency here behaves as a symptom before it counts as a cost — the visitor reads it as the system having stopped. Everything this section describes — the retrieval pipeline with its working-ack, the ladder that reroutes a dying lane, the co-browse session, the media roster — is a first-generation AI Orch in the §04 sense: a live coordination object with participants, dependencies, state and a lifecycle, written by hand today and generating the trajectories a learned layer trains against.

11.1  The floor is the primitive

The single most consequential finding in these notes is also the least visible. Injecting content into a live session while either party holds the conversational floor does not queue it or interleave it — it makes the model stop producing audio entirely. Two competing turns are arbitrated into silence. From outside it reads as a crash; the transcript shows an answer that simply never arrives.

So the layer maintains an explicit model of who owns the floor — nobody, the visitor, or the loop — inferred from transcription events, and injects only into a free floor. Because the floor is inferred, every latch expires on its own: a dropped event costs a pause, never the session. And past a deadline the loop deliberately treats its own view as stale and delivers anyway, on the principle that boundtalking over one syllable is recoverable and silence is not.

11.2  Never hold the answer

An action call in a real-time voice loop is a stall with a human attached. While a response is outstanding, nothing is generated — and our slowest actions were not slow by accident but by nature: resolving a video, fetching its transcript, or cold-starting a browser. Held serially, that produced tens of seconds of a voice that had simply stopped.

The fix is to make the response a receipt rather than a result. Slow actions answer immediately — working; the payload will arrive as a note — the conversation stays alive, and the real result is injected later through the floor-aware channel above. It is explicit that a receipt is not an answer and must never be described as one, because the obvious hazard of this design is inventing the result that was promised. Independently, the same instinct produced the earlier finding in reverse: the panel is painted before the action resolves, so the visual arrives while the voice is still mid-sentence rather than after it.

11.3  The visitor's sentence is state worth protecting

Sessions break. When they do, the naive queue drops microphone frames older than a few hundred milliseconds — correct while running, and exactly wrong during a reconnect, because the frames being discarded are the visitor's sentence. The queue is therefore two-mode: bounda short hold while healthy, a much longer one while the link is down, and an overflow policy that sheds audio but never control messages. What the visitor said survives the reconnection they never knew about.

12What didn't work

Knowing When to Speak published its negative results because they cost weeks. The same discipline applies here, and this list is the part of these notes we would most want another team to read.

12.1  A configuration flag that corrupted sessions after action calls

The most expensive failure of this period presented as everything at once: audio cutting off mid-word, the voice resuming somewhere else in its own sentence, and artifacts that never arrived. It looked like four bugs in four subsystems. It was one: a modality declaration in the session setup which is meaningful for cascaded speech pipelines and actively harmful on a speech-native path, where it corrupts the session — most often shortly after an action call, which is precisely when we were building visuals. Removing it eliminated the class. The transferable lesson is the shape: an option that is merely redundant on one architecture can be destructive on another, and real-time systems express that destruction as an intermittent behavioral fault rather than an error. We found it by reading close codes, not transcripts.

12.2  The system interrupted itself with its own sound

A gentle audio cue plays while a search runs. That cue leaves the speakers, re-enters the microphone, and — being sound arriving during the model's turn — was read by barge-in detection as the visitor beginning to speak. The system ducked and cut its own sentence, then resumed a few words later. Visitors experienced it as the voice stumbling over itself, which is exactly what it was.

The fix is small: a raised confidence threshold in a short window around each cue, so only a genuinely loud voice can take the floor there. The finding is not small. A full-duplex system must model its own output as adversarial input — not merely the synthesised voice, which every echo canceller already knows about, but every sound the product itself makes. Notification tones, transitions and ambient design are, to your own turn-taking detector, indistinguishable from a human.

12.3  A whitelist on a growing seam, and tests that proved nothing

Events travel from the panel to the loop through an intermediate frame. That hop forwarded a fixed list of fields chosen when the protocol was small. As the protocol grew, the list silently dropped the newest ones — including the action catalog itself, which meant the addressable surface was empty in every real session while our tests reported green. The tests injected messages after the hop.

Two rules came out of it. Never whitelist fields on a seam you expect to grow: forward the message and strip only what must not pass. And a test that bypasses a transport hop proves nothing about that hop — if a message crosses three boundaries in production, the test must cross three boundaries.

12.4  Behavioral drift moves; it does not sit still

Story mode weaves the wider technology landscape into conversation. Asked about agriculture, or a people's history, it kept steering back to technology — the wrong instinct entirely, and one we tried to fix four times. Each fix worked and the behavior reappeared somewhere else: first in the answer, then in the filler beat before the answer, then in the closing line that widens the lens, and finally — the one we did not expect — in the memory seed, because the summary of a previous session was itself full of that language and was being replayed as the opening of the next.

The lesson generalises past instruction-writing. A behavioral constraint must be applied at every seam that carries an example, because examples teach habits. And summaries feed forward: any system that compresses a session and re-injects it has built a channel through which today's drift becomes tomorrow's prior.

12.5  Three smaller ones, kept for the pattern

A cleanup routine deleted the browser profile while the browser was still exiting; the resulting error was thrown inside an interval callback and took down the whole service — an uncaught throw in a timer is a service-level failure, not a local one. A sandbox that escaped </script> in generated markup broke every artifact containing a script, because the payload was body markup and the escaping defended against nothing. And a null-origin sandboxed frame sends no cookies, so a reviewer gate quietly redirected every request for the artifact runtime — the security context that makes a sandbox safe also makes it anonymous, and anonymous is a state your own infrastructure may not recognise.

13What we measured, and what we only bounded

This section exists because the previous ones are full of numbers, and most of them are not measurements. Nearly every figure quoted above is a bound — a threshold, ceiling or timeout we chose and enforce. A bound tells you what the system will never exceed. It tells you nothing about what it does. We mark them differently for that reason, and we would encourage anyone reading engineering notes — ours included — to ask the same question of every number.

measuredThe latency campaign from Knowing When to Speak remains the only figure set gathered with a stated method: in-datacenter probes against the public production loop, text-injected turns, sequential, no exclusions, twelve sessions. Its headline results stand: a core turn reaching first audio in roughly 2.0–4.8 s, a live search returning in roughly 0.7–1.1 s, and a page read in roughly 2.3–2.6 s.

measuredThis period added per-action timing, reconnect duration, dropped-frame counters and a client playback-health beacon to the session record, so the next campaign can be run against instrumentation rather than a stopwatch. The spot figures we can stand behind are small-n and taken during development, not a campaign: an action that formerly held its response for tens of seconds returning a receipt in about one second; reconnect downtime around 1.4 s where the previous backoff alone could exceed four; and a browser session opening in roughly two seconds warm against fifteen to thirty cold. We state these as directional, with n in the single digits, because that is what they are.

boundThe thresholds: a four-second and twelve-second ceiling on held reveals; a ninety-second confirmation life; a settle delay before verification; two failures before a target is abandoned; a six-second floor between ambient operations and a per-session ceiling on them; one hero and two expanded artifacts on stage; a bounded page digest; two concurrent browser sessions; a short microphone hold while healthy and a longer one during reconnection; a delivery deadline past which the loop stops trusting its own view of the floor.

The verification suite is likewise a bound rather than a measurement: forty-two unit tests, of which the ones we value most are not the renderer cases but the refusals — that a vague reference is reported ambiguous instead of guessed, that a stale confirmation does not fire, that the sandbox policy names no wildcard, and that the instruction cannot advertise a component the client cannot draw.

Open problem — measuring a surface. We have no metric for the thing these notes are actually about. Latency we can measure; co-timing we cannot. There is no accepted measure of whether a visual arrived at the right moment relative to the sentence that explains it, nor of whether a person understood faster because it did. That gap is what §16 is for, and until it closes, claims about interactive intelligence — including ours — are architecture and taste rather than science. The instrument's first recorded runs now exist — the artifact-to-narration offset is measured in signed milliseconds per task — and every number lives on the bench page, never here, so the notes cannot drift out of date against their own ruler.

14Invariants added, and boundaries that moved

Knowing When to Speak stated six invariants and five boundaries. The surfaces described here add five invariants of their own, and — this is the part worth reading — two of the original boundaries have moved, which we would rather state plainly than let a reader discover by contradiction.

Boundaries that moved. B1 stated half-duplex output with no cooperative backchannel. It has partially moved: the loop now emits brief listening tokens while the visitor speaks. Simultaneous full-duplex speech remains out of scope, so B1 is narrowed rather than retired. B3 stated that memory dissolves at session end by design. That is now true only for visitors who are not signed in; with an account, sessions are retained and a compact summary seeds the next conversation. The privacy posture became a choice the visitor makes rather than a property of the system, which is a better design and a weaker guarantee, and §12.4 documents the first way it bit us.

New boundaries. B6 — the interactive surfaces described here run in staging, not in the public application. B7 — Lens is one page at a time, read-only, with a small concurrency ceiling; it is not a general web agent. B8 — the browser is not streamed; perception is change-gated stills, so motion within a page is invisible. B9 — the addressable surface is our own application only. Nothing here controls the visitor's device, other applications, or anything outside the panel.

15What generalisation requires

The first-generation Actra surface works, and it works because we wrote its ladder. The interesting question is what it would take for the ladder to be learned rather than written — which is the same as asking what would move these families rightward on Figure 14.

For an Actra: action selection over a published affordance space represented in the model rather than in policy code; reference resolution, ambiguity and confirmation as learned behaviors, evaluated against held-out surfaces the model has never seen; and transfer — a policy trained against one application's catalog performing against another's without new glue. The last of those is the load-bearing one, and it is the one our present catalog cannot demonstrate, because the catalog is ours. That is a statement about scope, not about capability: a specification derived from one surface is exactly how every interface standard has ever started.

For an Octra: routing decisions — which lane answers, whether to retrieve, when to speak, when to wait — produced by a model under a real-time constraint, and beating written policy on a benchmark that includes the floor. That benchmark did not exist, which is why §16 describes the one we have begun. The private half of the instrument already runs: an offline gate that replays recorded sessions through the live loop and adopts a changed policy only on strict held‑out improvement — candidate against incumbent, any regression a veto. §16 is that discipline made public.

How the ladder starts to be learned is worth stating precisely, because it is the part of the program that keeps the rest honest: weights are earned through the gate, not written into it. Every session the surface serves leaves a trajectory — what was asked, what was staged, what was acted on, what verified, what failed — and that ledger is the families' native corpus: not text scraped from the world but the interaction loop itself, recorded from the inside. Candidate behaviors are evolved offline, replayed against held-out sessions, and admitted only on strict improvement through the gate described above, in which execution metrics hold a veto and any regression collapses the candidate back to the shipped behavior. Nothing self-modifies in production. The hand-written ladder is therefore the trained families' curriculum rather than their rival — the decisions it takes today are the labels they train against.

Alongside this work, the LAIMA classes — LaLaMo, LaMimo and Lasamo — are in research and development at Labsintelligence. They are hierarchical forms, not further families: LaLaMo, LaMimo and Lasamo name the three scales a Labs model ships as, and the two families of these notes take all three — an Actra or an Octra at LaLaMo class is the frontier form of the same family whose LaMimo and Lasamo forms serve the working and light deployments. The perspective-sphere capabilities that remain — flow detection, the collaborative matrix, spatial world-building — are the same registry-tense roadmap (see Interactive Perspective Spheres). The family × form map, with the shipped system placed where it actually is and every unreleased rung marked as research, is kept on the announcement page: Introducing Actras & Octras.

16Scoring it: Interactive Intelligence Benchmark

Every claim in these notes is currently unfalsifiable by anyone outside Labs, and that is not a position we want to hold. The existing public benchmarks for real-time voice are good and we use them, but none of them measures what these notes are about. VoiceBench, URO-Bench, SD-Eval and Big Bench Audio score comprehension. Talking-Turns and Full-Duplex-Bench score timing and turn-taking. EchoChain scores state revision under interruption. τ-Voice, the strongest of them, scores grounded task completion over a duplex channel^[17,18,19]. All of them score the audio channel and, at most, whether a call was emitted.

None of them asks whether the artifact arrived on time, whether the referent resolved, or whether the agent told the truth about what it did.

So we are publishing Interactive Intelligence Benchmark — IIB‑1 — with its tasks, its rubric, its scoring and its session recordings — including a truthfulness penalty that scores a false claim of success below a frank admission of failure. It is named for the category rather than for us, and any team named in it is invited to submit measured runs.

Two composite readings are defined over its five tracks, one per family, because the tracks measure two different competences. The Actra Score is the weighted mean of the action-side tracks — co-timing, reference, verification, shared surface — at their published weights, renormalized; the Octra Score is floor discipline, the Speech‑to‑Orchestration competence. Both recompute from the published per-track scores, so a reader who disagrees with the weighting can re-derive their own; and a composite with a track missing renders as pending rather than as a smaller number, because a hole in the basis is not a measurement.

References

1. Sesame. Crossing the uncanny valley of conversational voice. sesame.com/research, 2025.
2. Défossez, A., et al. Moshi: a speech-text foundation model for real-time dialogue. arXiv:2410.00037, 2024.
3. MiniMax. MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encoder. arXiv:2505.07916, 2025.
4. Thinking Machines Lab. Interaction Models. thinkingmachines.ai/blog, 2026.
5. Skantze, G. Turn-taking in Conversational Systems and Human-Robot Interaction: A Review. Computer Speech & Language, 2021.
6. Nguyen, T. A., et al. Generative Spoken Dialogue Language Modeling (dGSLM). arXiv:2203.16502, 2022.
7. Zhang, Q., et al. OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation. arXiv:2410.17799, 2024.
8. OpenAI. GPT-Live System Card. deploymentsafety.openai.com/gpt-live, July 2026.
9. Google. A2UI: declarative, streamed user interface across trust boundaries. Developer documentation, 2026. — streaming generative UI as a protocol rather than a rendering trick.
10. Vercel. streamUI / AI SDK. sdk.vercel.ai, 2024–2026. — components streamed during generation.
11. Microsoft. Playwright MCP. github.com/microsoft/playwright-mcp, 2025–2026. — accessibility-tree snapshots rather than pixels as the agent's action surface.
12. Abuelsaad, T., et al. Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems. arXiv:2407.13032, 2024.
13. Brohan, A., et al. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv:2307.15818, 2023.
14. Kim, M. J., et al. OpenVLA: An Open-Source Vision-Language-Action Model. arXiv:2406.09246, 2024.
15. Black, K., et al. π₀: A Vision-Language-Action Flow Model for General Robot Control. arXiv:2410.24164, 2024.
16. Vaswani, A., et al. Attention Is All You Need. arXiv:1706.03762, 2017.
17. τ-Voice: Evaluating full-duplex voice agents on grounded tasks. arXiv:2603.13686, 2026.
18. Lin, D., et al. Full-Duplex-Bench (v1–v3). github.com/DanielLin94144/Full-Duplex-Bench, 2025–2026; v3 arXiv:2604.04847.
19. Chen, Y., et al. VoiceBench: Benchmarking LLM-Based Voice Assistants. arXiv:2410.17196, 2024.