Pilotfish: Rethinking how agents browse
Fewer token-heavy observations, fewer model round trips and evidence-based waiting where a page allows them, and the points where each shortcut stops.
Illustrative workflow: the portal, customer and figures are fictional.
Acme Fasteners owes $8,400.00 on overdue invoice INV-1042, and a partial payment posted this morning. Before anyone chases the rest, an agent in the receivables portal has to:
- match the payment to the invoice;
- read the refreshed outstanding balance, not the stale list figure;
- check that the invoice is neither disputed nor already covered by a recent reminder;
- prepare a reminder draft with the confirmed amount for a reviewer, and stop before Send.
The draft depends on every one of those checks.
A naive browser loop makes that care expensive. Each move between the invoice list, payment activity and reminder history can resend a mostly unchanged table. A page that has gone quiet can still show the interim balance under a "Refreshing balances" bar. And once the checks pass, each known draft field can still cost its own model decision.
Pilotfish, an open-source browser MCP, rethinks what passes between the browser and the model: carry forward the context already delivered, send the facts that changed, wait for evidence that the page is ready, and batch only the next steps that are already known. The judgment about the invoice stays with the model and the reviewer. The sections below follow this job through each mechanism, including where it stops saving.
Print the page the way a model reads it
Before deciding what to resend, decide what a page is. The portal's version is markup built for rendering. The model needs to know what each control and value is, what it is called, what it holds, and how to point at it.
<tr class="row_3kq7 is-overdue" data-id="9f31"> <td class="cell_a1">Acme Fasteners</td> <td><a class="lnk_7" href="/invoices/INV-1042">INV-1042</a></td> <td class="cell_dt">Oct 2</td> <td><span class="badge_x2 red">Overdue</span></td> <td class="num">$8,400.00</td> </tr>
Acme Fasteners | [e5]INV-1042 | Oct 2 | Overdue | $8,400.00e5. Dropped: classes, the data attribute and the wrapper span. One row, one line.Pilotfish's serializer prints one line per meaningful element in an HTML-like grammar: tag or role, name, value and state, plus a short ref such as e5 that the model uses to act and that stays the same from one snapshot of the page to the next.
A plain table row prints on one line with its cells in column order between bars, so an amount stays with its invoice and its column. Styling classes, generated ids, attributes that only point at other elements and empty tracking frames are left out. The compact options are written to keep everything a reader acts on or quotes.[1]
Two more choices keep each observation small:
- The visible screen first. A reply to an action covers the visible screen rather than three screens of results, and ends with a count of what sits above and below it.
- A named way to read more. When a printout is cut at the character limit, its last line names the ways to read the rest:
browser_scroll,browser_get_text, orbrowser_snapshotwithwhole_page.[2]
Offscreen content is not declared irrelevant. It is just not prepaid.
Send only what changed
Take the agent's first move: it types "Acme" into the Customer filter. The field takes the value, two matching customers appear, a "Recent" link steps aside, and the invoice table does not change. With the screen in compact form, the reply can be a list of changes. The real question is what the changes are measured against.
Pilotfish measures them against the delivered base: the last whole screen the caller actually received. Not the server's latest capture and not a cached copy, but the page that reached the model.
Because the model has that page in context, a change list against it is something the model can apply: lines not listed are as shown there, and their refs still work.[3]
Step through the first part of the task:
One receivables page, four replies
Reply 1 carries the whole visible screen, invoice table included. Once it is delivered, it is the base.
- Changed
- New
- Unchanged context, marked ~
- Removed
- Not resent
Example page. Line formats follow the serializer, change-list and evidence code.[1][4][5]
What makes the list safe to apply
- Page order. A re-sorted invoice table, or two rows that swapped amounts, reads as a change in its new place, never as the same page.
- Context. Each run of changes follows the unchanged line before it, marked
~, so the model knows where the run sits. - Removals. Interactive elements that left the page are named in a closing
GONEline, so the model is not left holding a ref to something that no longer exists.[4]
When a full screen is the better reply
A change list is only worth sending when it is shorter and clearer than the screen. Pilotfish sends the whole screen, and makes it the new base, when:[3]
- the document or address changed, since the old lines describe a page that is gone;
- the base is more than eight browser results old;
- the change list would be longer than half the screen;
- a page was cut at the character limit, or contains text that looks like the list's own markers, such as a line reading
~ 2 hrthat could pass for context and put later lines under the wrong rows.[4]
None of these is the optimization failing. They are where it stops being one.
Delivered means delivered
Pilotfish also has an output guard: a reply that would show a configured secret value is withheld whole rather than partly masked. The guard covers values it can recognize, not every possible secret.[10]
That means the server can capture a screen that the model never receives.
Predict
The server captured the invoice page, but the guard withheld that reply because it would have shown a configured secret value. The next click changes one field. What should the next reply be?
A base commits only when its reply is delivered. A withheld reply leaves no base, and until a whole screen reaches the model again, every action prints its page in full.[3]
Wait for evidence, not silence
The agent picks Acme Fasteners, the filtered list loads, and it opens INV-1042. The invoice page draws quickly, but its balance panel refreshes on its own to apply the new payment. Before replying, pilotfish has to decide when the page is ready to describe.
A fixed sleep is wrong both ways: too short and the model reads a half-drawn page, too long and every action pays for the slowest one.
Waiting for the whole page to go quiet fails differently. Analytics, ads and polling keep many pages busy indefinitely, so a page-wide quiet window tends to run to its cap.
What the settle step watches
It watches only what the action started. For a short window after the action, longer after typing into a debounced input, it looks for a request, a navigation or a DOM change. Then:
- Nothing started: the reply goes now.
- Only the DOM changed: it waits for the changes to stop.
- The action sent requests: it waits for those, including the server's answer to any write, then for a short render window.
The result records which conditions were met.[6]
Quiet is not the same as ready
The settle step answers "has the action finished?" It does not always answer "is the page ready?"
Predict
After the click on INV-1042, the request has been answered and the DOM has stopped changing. The balance panel shows $8,400.00 outstanding under a bar that reads "Refreshing balances". Is that amount ready to report?
The evidence behind a reply
Action click INV-1042
After an action, pilotfish also looks for the page's own loading signals: an indeterminate progress bar, aria-busy, or a progress element labelled loading, fetching, searching or updating. It waits up to six seconds for them to clear.
If one is still there, the reply says the page is still loading instead of presenting interim content as final.[7] A write that never got an answer is handled the same way: reported as unanswered, not assumed accepted.[6]
Read the answer, not the page
The balance has refreshed. The next decisions need a few facts from the invoice page: its ID, the current outstanding amount, whether it is disputed and when the last reminder went out. Printing every line item, payment and activity entry to read four values is the same repetition in a new place.
browser_query runs read-only JavaScript in a world of its own, with page writes blocked, and returns only the values asked for. For this invoice, that can be as small as:
Example values{ "invoice": "INV-1042", "outstanding": "$5,150.00", "disputed": false, "lastReminder": null }
The $5,150.00 is the balance after the partial payment, which the list had not caught up with. A comment in the source records the case that motivated the tool[2]:
Smaller replies to actions
Acting tools have a related switch. With snapshot:false, the printed page is left out of the reply while the checks, the action evidence and the write receipts still run.
With escalation on, an unmet or ambiguous expectation, or an event such as a dialog, a new tab, an invalid field or a page still loading, brings the page back.[8]
The catch: a small reply is not free if the model then needs another read to finish the thought. The measure is the whole task, recovery reads included.
Decide once, check every step
Last stretch. By now the balance is confirmed at $5,150.00, the invoice is not disputed and no recent reminder has gone out, so a draft is appropriate.
Had the agent found a dispute or a recent reminder, this workflow would stop for review instead of drafting. That rule belongs to the workflow, not to the MCP.
With those checks settled, the recipient's email, the subject, the message and an internal note for the collections team are all known before the first keystroke. Seeing the email typed tells the model nothing new about the subject.
Pressing Send is different. It is a consequential action that waits on a person's review, and this workflow stops before it.
Known steps and uncertain steps
- Known: no earlier result could change the step. Known steps can share one decision.
- Uncertain: the step depends on what the page does, or on a decision that needs review. Uncertain steps need a fresh look.
browser_actions takes up to ten known steps in one call and runs them in order. Each step gets the same checks it would get as its own tool call, and the reply has a line per step with the facts that must not be lost, such as what a field read back, followed by the last page as changes against the base.[9]
What a batch removes, and what it keeps
Three counts are worth keeping apart:
- Model call: the model reading and deciding once.
- Tool call: one request to the browser tool.
- Browser step: one action on the page.
Batching keeps every browser step and its checks; what it removes is the model calls and tool calls between them. The steps still execute and settle one by one, and they are not merged into one request to the site.
When a batch stops
Predict
The portal checks the recipient's address as it is typed, and the check is still spinning after the email step. Subject, message and note are known values. Should the batch type them?
Four known fields, two ways
One call per step
The model reads a result before each step.
One batch
One call, then checked steps in order.
- Tool call from the model
- Result back to the model
- Next step with no model call
The batch reply says: Stopped after step 1: the page was still loading (later steps would act on a page still changing); steps 2-4 were not run.
A batch ends:[9]
- at the first step that fails;
- after a step that moves the page to another address, since later refs would belong to the old page;
- after a step that raises a dialog, opens a tab, has a server write refused or unanswered, leaves the page still loading, or leaves a check or sign-in on the page.
One thing does not end it on its own: a field the page marks invalid. That is reported on the step's line, and the model sees it in the reply.
In this workflow a finished batch leaves four filled fields and nothing else. Send is not part of it, and it is not taken here at all: the draft waits for a person.
What the published runs show
The mechanisms say where savings should come from. Whether they show up across real tasks is a separate question, and the project keeps a measurement of it.
The Pilotfish repository includes a rerun of LangChain's WebVoyager comparison setup, with its tasks, prompts and judge criteria, putting the pilotfish open MCP beside Playwright MCP, the baseline that comparison used. Both arms ran on the same machine against the same live pages at the same time.[11]
The tasks are WebVoyager's public web tasks; the invoice workflow above is only an illustration.
Paired WebVoyager rerun, means per task
Arithmetic means per task over the tasks that ran, excluding errors before the first model turn. Model cost includes cached-token pricing. The same model drove and judged both arms, and timings were recorded with five tasks running concurrently.[11][12]
Read the chart with its limits:
- Correctness was slightly higher for pilotfish in this run, which does not establish that the two are equal or that either is better.
- These are the project's own published runs, not third-party verified scores.
- They compare whole systems rather than isolating any mechanism above.
The runner, the task file and every result row are in benchmarks/webvoyager-100.[11][12]
The tradeoff
Pilotfish bets that a compact default, plus deliberate ways to see more, beats one complete representation of every page. That is a real tradeoff.
A compact observation can leave out the control that mattered, and some tools let the model choose its own observation strategy, which is a legitimate design.
My answer is that each shortcut here carries its way back: the whole screen when the base no longer fits, a stop when the page is still changing, a loading note instead of an interim answer, a full snapshot one call away. The design explains where the savings come from. Whether accuracy holds on your workload is something to measure.
Try it
The open MCP runs on your own machine and drives a local Chrome or Chromium. It needs Node.js 22 or later, and no Pilotfish account or API key. In Claude Code:
claude mcp add pilotfish -- npx -y @gaiventures/pilotfish-mcp
Setup for other MCP clients is in the open MCP docs. This is the local MCP, not the managed service, and the TypeScript SDK is a separate package.
- Repositorygithub.com/gAI-ventures/pilotfish-ai
- Open MCP docsSetup for other clients
- npm: the open MCP@gaiventures/pilotfish-mcp
- npm: TypeScript SDK@gaiventures/pilotfish, separate from the MCP
Point it at a page your agent has already read and look at what comes back. If something breaks, an issue on the repository helps, and so does a star if you find it useful.
References
The implementation references point at one public commit of gAI-ventures/pilotfish-ai, b990f0f, so line numbers stay put. They show what the code does, not how well it performs. The measured references hold the benchmark method and rows behind the chart.
Implementation, at commit b990f0f
- [1]Serializer grammar and compact optionspackages/
perception/ src/ serialize.ts, lines 1-78; packages/ perception/ src/ serialize.ts, lines 301-376The one-line-per-element grammar, refs, and the compact options that drop generated ids and pointer attributes, limit a printout to the screen, and skip empty tracking frames; and plain table rows printed on one line, cells in order between bars. - [2]MCP session defaultspackages/
mcp/ src/ session.ts, lines 185-223What the MCP switches on: compact screens, visible-screen replies with above and below counts, the read-more hint, the loading wait, browser_query with its 20,000 and 2,900 character note, and replies withheld whole when they would show a secret value. - [3]Delivered base and change-list rulespackages/
mcp/ src/ server.ts, lines 325-438pageText: a base commits only on delivery, a withheld reply leaves no base, and a whole screen replaces a change list for another document or address, a base more than eight results old, or a change over half the screen. - [4]Ordered change-list formatpackages/
perception/ src/ observation.ts, lines 142-263Page order, ~ context lines, replaced-line counts, the closing GONE line, and the whole screen when a page was cut or contains text that looks like the list's markers. - [5]Replies when nothing changedpackages/
perception/ src/ evidence.ts, lines 270-271; packages/ agent/ src/ agent.ts, lines 3862-3875The one-line evidence for an action with no observable effect, and the rule that no page is printed then. - [6]Evidence-based settlingpackages/
browser/ src/ page.ts, lines 1154-1255settleFast: the short watch window, the longer one for debounced inputs, waiting for caused requests and write answers, ignoring other traffic, and recording which conditions were met. - [7]Waiting on loading indicatorspackages/
agent/ src/ agent.ts, lines 265-272The loading indicators checked, the six-second limit, why a quiet network and DOM are not enough, and the interim-price case that motivated it. - [8]Reply-size optionspackages/
mcp/ src/ server.ts, lines 57-83; packages/ agent/ src/ agent.ts, lines 575-580snapshot:false, the opt-in expectation, and the escalation conditions that bring the page back. - [9]Batched steps and stop rulespackages/
mcp/ src/ server.ts, line 181; packages/ mcp/ src/ server.ts, lines 456-457; packages/ mcp/ src/ server.ts, lines 766-845The ten-step limit, the step facts that end a batch, and browser_actions itself. An invalid field is reported on its step's line but is not a stop condition. - [10]Output guardpackages/
mcp/ src/ server.ts, lines 593-708; packages/ agent/ src/ secrets.ts, lines 61-97How a reply that would show a secret value is withheld whole, and when screenshots are refused. It covers values the guard can recognize.
Measured runs, at the same commit
- [11]WebVoyager-100 rerun: methodbenchmarks/
webvoyager-100/ README.md, READMEWhat was copied from LangChain's setup, the model that drove and judged both arms, five tasks running at a time, list prices including the cached-token rate, and the exclusion of tasks that errored before the first model turn. Its own table reports medians; the means in this post come from [12]. - [12]WebVoyager-100 rerun: result rowsbenchmarks/
webvoyager-100/ results/ 2026-11-11-luna.jsonl, results/2026-11-11-luna.jsonlOne row per task and arm, with tokens, tool calls, seconds and cost. The chart's means are computed from these rows.