the engineering log

Take Over, Hand Back

Skyflo 1.4 lets you and an agent share one browser and one terminal. The hard part was not letting a person grab the mouse. It was deciding who owns the surface at every instant, what the agent may see of your turn, and where the mission resumes when you give it back.

Read the full page:How Skyflo runs a mission, and learns from it
·9 min read·co-controlbrowserterminalvoice

The Moment a Mission Needs a Person

Every long mission has a moment the agent cannot get past on its own. The staging console wants a second factor. The vendor portal needs the card that lives in your wallet. The cluster you asked it to inspect is behind a login that was never meant to be scripted, and should not be. Until now, that moment ended the useful part of the mission. You did the step somewhere else, in your own browser or your own shell, and then had to explain to the agent what the world looked like now.

Skyflo 1.4 removes the somewhere else. The browser pane and the terminal pane an agent is working in are the same panes you can take. Press ⌥⌘C, or ⌥⌘T for the shell, or click Take control on the bar above the pane, and the agent's clicks and keystrokes stop that instant. The page stays on the page it was on. The shell keeps the session it was in, with its history and its working directory. You do the part that needed a person, and when you press Continue with Skyflo, the mission picks up from what is on screen now.

This sounds like a small feature, and the code that lowers the input shield is small. Everything around it is where the release went.

One Owner, Always

The first design rule is that a surface has exactly one owner at any instant, and the answer to who owns it is never derived from a guess about what is happening. We learned this the expensive way in the first build.

The browser originally kept a single field for both the agent's claim on the pane and the person's hold on it. Take control while a turn was running, let the turn end while you still held the page, and the claim was left behind: every later attempt by any mission to acquire the browser was refused, for a turn that no longer existed. The pane was wedged for the rest of the session. They are two facts and they are now two fields. A turn ending always ends the claim. A hold outlives it, so a mission finishing while you are halfway through a sign-in neither drops you back into an agent-driven page nor blocks the next mission for good.

The terminal had the mirror image. Take control and press Continue in quick succession and the two transitions interleaved, because taking control can wait on a session being created. The bar would say you were driving while the write gate said the agent was. Every ownership transition and every write to the shell now runs on one queue, so the last press wins, and the chrome and the gate cannot disagree.

The bar itself was rebuilt around that rule. It says one of four things: Skyflo is using the browser, you have the browser, the reason it is waiting for you, or that it is handing the browser back. It offers one action at a time. Earlier versions narrated the button next to them and named the mechanism in brackets, which told you nothing you could act on. Whether the thing being handed over is a page or a shell is not a difference anyone acts on either, so it is one component and only the noun changes.

The Handback Reads the Real Page

The second rule is that a handback resumes from what is true, not from what the agent remembers.

When you press Continue, Skyflo raises the shield first, then reads the page: its title, its address, and what it can see. Only then does the turn resume, with that reading in front of it. The order matters. Reading first and shielding second left a window in which a navigation you had already started could land after the read, and the mission would carry on against a page that had just gone. So the shield goes up, and if you clicked a link a moment before pressing Continue, the handback waits for that navigation to settle, for at most two seconds. A page that never finishes must not hold the handback open, and whatever is on screen when the wait is up is the truth, so that is what gets reported.

The terminal does the same with its last lines of output. If you ran the migration yourself, the agent's next step begins from the output of the migration, not from its plan to run one.

A handback that cannot read the page aborts and leaves the pane with you. A handback that reads the page but cannot restart the mission says so separately, because you are somewhere different in each case and the next safe action is different too.

Your Turn Is Yours

The third rule is that the agent does not get a transcript of your turn.

Keystrokes typed into the shell while you hold it are excluded from what the agent can pull. The output it reads is bounded, pull-only, and clears whenever ownership changes hands, so it starts from the handback rather than from a buffer of your session. Before any terminal output reaches a tool result, escape sequences and credential-bearing addresses are filtered out of it. On the browser side, an access token left in a URL fragment after a sign-in never reaches the model. We built an Electron harness that drives real WebContents views, a real input shield and a real pseudo-terminal through exactly these boundaries, because a fake view commits synchronously and a real one does not, and the difference is where two of the bugs above were hiding.

This is a filter with known shape, not a general secret detector, and we say so in the product's own documentation. It removes the classes of leak the design creates. It does not read your mind about what else on the page is sensitive.

Agents Can Ask

The counterpart to taking control is being asked for it. When an agent reaches a step only a person can do, it yields the surface rather than driving on. The bar shows the agent's reason, holds the pane for you, and waits.

Asking used to be theatre. The earlier request-help tool told the model it had yielded the session and paused. It had done neither: it attached a notice to a pane the agent kept operating. On the terminal it was worse, silently demoting a person who had already taken the shell and accepting the next agent write while the chrome still said you were driving. Asking for help is now a real yield on both surfaces, and a help request can never take the shell off a person who is holding it.

Saying It Out Loud

In 1.2 we argued that a voice is a control surface: a fixed grammar claims the sentences that are the app's own controls, and everything else is work for the harness. Co-control is the first time that split had to depend on state.

"Take control of the browser" and "your turn" are now controls, and they do exactly what the bar does. But "continue", "done" and "I'm done" are also ordinary words, and the first build caught the language along with the intent. "Continue reading the file and tell me what it says" became a handback carrying the rest of the sentence as a note, and a matched control never reaches the model, so the request was not misunderstood. It was eaten. Worse, "I'm done" was reaching the farewell reader and hanging up the call, which is the worst available response to somebody who has just finished a sign-in and wants the mission to carry on.

The reading now depends on whether you are holding a surface. Holding the browser, "done" is a handback. Holding nothing, it is a word, and it goes to the harness with the rest of the sentence. "I'm done" ends the call only when nothing is being held. And "start a terminal session" opens the shared shell before any work begins, so you can direct it live rather than describe it in advance.

The App Got Faster on the Way

The same release carries the Desktop performance program that ran alongside it: sixty-odd changes, accepted in five cycles, each cycle measured against the last on a fixed workload and admitted only with no metric regression left unexplained. Nothing in it changed what Skyflo does. All of it changed how much work Skyflo does to do it.

The local runtime used to compile every schema validator it owned before it would listen, about eight hundred milliseconds on a modern machine, for validators a launch never uses. It now compiles them when first needed, warms the handful an ordinary turn appends in idle time after it is listening, and keeps its compiled code between launches. It starts while the process runner is coming up, held at a gate, rather than after it. The shell window appears without waiting for saved state to load.

An idle Skyflo also does less. The mission list was rebuilt on every two-second poll, and every poll re-rendered the sidebar. The list is now built once and served with a revision; the app asks whether that revision is still current and, when it is, nothing is sent and nothing redraws. Hovering a mission row used to run a process sample on the main thread for every row you crossed. Quitting waited on a poll timer after the helpers had already gone.

On the workload the program measured, in development builds, cold start to interactive fell from about 2.8 seconds to under one, idle CPU with thirty missions loaded dropped by three quarters, memory across all processes came down by about a hundred megabytes, and quitting takes less than half the time. Those figures were taken on Linux, unpackaged, which is why they are in this paragraph and not in the release notes. The notes carry what we measured on the shipped Mac build, and we will keep publishing the numbers as they are rather than as we would like them.

Where It Stands

Co-control is in Skyflo 1.4 on every plan, including Free; the browser and the terminal it shares are the ones the app always had. Voice Mode, which reaches the same controls by name, still needs a paid plan, and the pricing page lists what each plan carries. Everything else in the release is in the 1.4.0 release notes.

The idea is the one every release since 1.0 has been built on. The harness owns the work and records what happened. A person can step in, and stepping in is now a first-class state with one owner, a clean boundary, and a truthful handback, rather than a workaround in another window. The agent does not lose the thread when you take the wheel, and it does not get to watch you drive.