Notebooks · Reliability and control · 59

BrowserAgentTool — ComputerUseAgent inside the main Agent (v1.0.8)

Name: SHIPIT Agent
Author: SHIPIT

The killer pattern: your main Agent plans, and when it needs to drive a browser it delegates to a sub-ComputerUseAgent via a single browser_use tool call.

Open in Colab View on GitHub 6 code cells · 4 with saved output

Its imports and arguments are checked against shipit-agent 3.0. The outputs were saved when it was written, for an earlier release.

In [1]

python

from pathlib import Path
import sys
ROOT = Path.cwd().resolve().parent if Path.cwd().name == 'notebooks' else Path.cwd().resolve()
if str(ROOT) not in sys.path:
    sys.path.insert(0, str(ROOT))

from shipit_agent import Agent
from shipit_agent.computer_use import (
    BrowserAgentTool, MockBrowserSession,
)

Example 1 — Plain Agent + browser tool

We build a BrowserAgentTool against a MockBrowserSession (so the notebook runs offline). In production, swap the factory for PlaywrightBrowserSession.launch(...).

In [3]

python


class ScriptedLLM:
    def __init__(self, replies):
        self.replies = replies
        self._i = 0
    def complete(self, *, messages, **_):
        from shipit_agent.llms.base import LLMResponse
        text = self.replies[min(self._i, len(self.replies) - 1)]
        self._i += 1
        return LLMResponse(content=text)

class StringLLM:
    """For the sub-agent — accepts string complete() returns."""
    def __init__(self, replies):
        self.replies = replies
        self._i = 0
    def complete(self, *, messages, **_):
        text = self.replies[min(self._i, len(self.replies) - 1)]
        self._i += 1
        return text

In [4]

python

# The browser tool's sub-agent uses a vision-capable LLM (StringLLM here)
browser_tool = BrowserAgentTool(
    llm=StringLLM(replies=[
        'ACTION: navigate https://apple.com',
        'ACTION: click 240,80',
        'ACTION: done The iPhone 15 Pro starts at $999.',
    ]),
    browser_factory=lambda: MockBrowserSession(),
    max_iterations=8,
)

# Direct invocation of the tool — no main agent yet
result = browser_tool.run(goal='Find the iPhone 15 Pro starting price.')
print(result.text)
print()
print('metadata:')
for k in ('status', 'iterations'):
    print(f'  {k}: {result.metadata[k]}')

Out

Browser sub-agent finished (status=done, iterations=3).
Goal: Find the iPhone 15 Pro starting price.

Answer: The iPhone 15 Pro starts at $999.

Last actions:
  · iter 0: navigate
  · iter 1: click
  · iter 2: done

metadata:
  status: done
  iterations: 3

Example 2 — Main Agent calls `browser_use` as one tool among many

The main agent decides when to use the browser vs simpler tools (web_search, pdf_extract, etc.). The BrowserAgentTool shows up alongside them — same Tool protocol.

In [5]

python

browser_tool_2 = BrowserAgentTool(
    llm=StringLLM(replies=[
        'ACTION: navigate https://example.com/contact',
        'ACTION: type "Inquiry from research bot"',
        'ACTION: done Form filled and submitted.',
    ]),
    browser_factory=lambda: MockBrowserSession(),
    max_iterations=6,
)

# Toolset visible to the main agent
agent = Agent(
    llm=ScriptedLLM(replies=['I will respond directly.']),
    tools=[browser_tool_2],
    auto_use_skills=False,
)

print('Tools the main agent sees:')
for t in agent._effective_tools('any'):
    print(f'  - {t.name}: {t.description[:80]}...')

print()
print('Tool schema (what the model sees):')
import json
print(json.dumps(agent._effective_tools('any')[0].schema(), indent=2))

Out

Tools the main agent sees:
  - browser_use: Drive a real browser to accomplish a goal that requires reading and interacting ...

Tool schema (what the model sees):
{
  "type": "object",
  "properties": {
    "goal": {
      "type": "string",
      "description": "What the browser sub-agent should accomplish, in plain English. Be specific about what to extract or do."
    }
  },
  "required": [
    "goal"
  ]
}

Example 3 — `share_browser=True` for multi-step browser sessions

When a workflow needs multiple browser interactions that share state (logged-in session, cookies, scroll position), enable share_browser=True. The same browser is reused across tool.run() calls until you tool.close() it.

Trade-off: faster (no relaunch) but state leaks between calls — only enable when the parent agent is aware that browser state persists.

In [6]

python

browsers_made = []
def factory():
    b = MockBrowserSession()
    browsers_made.append(b)
    return b

shared_tool = BrowserAgentTool(
    llm=StringLLM(replies=[
        'ACTION: done step 1 finished',
        'ACTION: done step 2 finished',
        'ACTION: done step 3 finished',
    ]),
    browser_factory=factory,
    share_browser=True,   # ← reuse the browser across calls
)

for goal in [
    'Log in to the dashboard',
    'Open the billing page',
    'Download the latest invoice',
]:
    out = shared_tool.run(goal=goal)
    print(f'  → {out.text.splitlines()[0]}')

print()
print(f'browsers created: {len(browsers_made)}  (would be 3 without share_browser)')

# Always close at the end
shared_tool.close()
print('shared browser closed')

Out

  → Browser sub-agent finished (status=done, iterations=1).
  → Browser sub-agent finished (status=done, iterations=1).
  → Browser sub-agent finished (status=done, iterations=1).

browsers created: 1  (would be 3 without share_browser)
shared browser closed

Production setup

The same code, but with Playwright + a real LLM:

# pip install playwright
# playwright install chromium

from shipit_agent import Agent, VerifierNetwork
from shipit_agent.computer_use import (
    BrowserAgentTool, PlaywrightBrowserSession,
)

browser_tool = BrowserAgentTool(
    llm=opus_llm,                                  # vision-capable
    browser_factory=lambda: PlaywrightBrowserSession.launch(headless=True),
    max_iterations=12,
)

verifier = VerifierNetwork(llm=haiku_llm, goal='Research only — no purchases.')

agent = Agent(
    llm=opus_llm,                                  # main planner
    tools=[browser_tool, WebSearchTool(), PDFTool()],
    verifier=verifier,                             # gate destructive actions
)

result = agent.run(
    'Find the cheapest direct SFO-JFK flight on May 20 '
    'and summarise the booking page.'
)

Why this is powerful

Composable — browser_use is just a Tool. Add it to any Agent. Stack with RAG, structured output, verifier, memory consolidation.
Gated — pair with VerifierNetwork and the verifier vetoes destructive browser actions before they run.
Observable — every browser action lands in result.metadata['actions'], which flows through the parent agent's events.
Self-hosted — no Devin/Operator subscription. Your LLM, your infra, your data.

→ See Agent → ComputerUseAgent for the full reference.

from pathlib import Path import sys

ROOT = ( Path.cwd().resolve().parent if Path.cwd().name == "notebooks" else Path.cwd().resolve() ) if str(ROOT) not in sys.path: sys.path.insert(0, str(ROOT))

from examples.run_multi_tool_agent import build_llm_from_env from shipit_agent import Agent

!playwright install chromium

In [7]

python

!playwright install chromium

Out

Removing unused browser at ~/Library/Caches/ms-playwright/chromium-1223
Removing unused browser at ~/Library/Caches/ms-playwright/chromium_headless_shell-1223
Downloading Chromium 133.0.6943.16 (playwright build v1155) from https://cdn.playwright.dev/dbazure/download/playwright/builds/chromium/1155/chromium-mac-arm64.zip
123.3 MiB [                    ] 0% 0.0s123.3 MiB [                    ] 0% 100.8s123.3 MiB [                    ] 0% 58.0s123.3 MiB [                    ] 0% 66.7s123.3 MiB [                    ] 0% 50.5s123.3 MiB [                    ] 0% 50.3s123.3 MiB [                    ] 0% 28.9s123.3 MiB [                    ] 0% 30.0s123.3 MiB [                    ] 0% 22.5s123.3 MiB [                    ] 1% 14.4s123.3 MiB [                    ] 1% 13.2s123.3 MiB [                    ] 2% 10.0s123.3 MiB [                    ] 2% 11.0s123.3 MiB [=                   ] 2% 9.2s123.3 MiB [=               … (3,454 more characters)
Chromium 133.0.6943.16 (playwright build v1155) downloaded to ~/Library/Caches/ms-playwright/chromium-1155
Downloading Chromium Headless Shell 133.0.6943.16 (playwright build v1155) from https://cdn.playwright.dev/dbazure/download/playwright/builds/chromium/1155/chromium-headless-shell-mac-arm64.zip
77.9 MiB [                    ] 0% 0.0s77.9 MiB [                    ] 0% 65.5s77.9 MiB [                    ] 0% 43.8s77.9 MiB [                    ] 0% 28.8s77.9 MiB [                    ] 0% 18.0s77.9 MiB [                    ] 1% 12.0s77.9 MiB [                    ] 1% 11.1s77.9 MiB [                    ] 2% 8.9s77.9 MiB [=                   ] 2% 6.6s77.9 MiB [=                   ] 3% 5.6s77.9 MiB [=                   ] 4% 4.9s77.9 MiB [=                   ] 7% 3.3s77.9 MiB [==                  ] 9% 2.7s77.9 MiB [==                  ] 12% 2.2s77.9 MiB [===                 ] 14% 1.9s77.9 Mi… (1,594 more characters)
Chromium Headless Shell 133.0.6943.16 (playwright build v1155) downloaded to ~/Library/Caches/ms-playwright/chromium_headless_shell-1155

Notebooks · Reliability and control · 59

BrowserAgentTool — ComputerUseAgent inside the main Agent (v1.0.8)

The killer pattern: your main Agent plans, and when it needs to drive a browser it delegates to a sub-ComputerUseAgent via a single browser_use tool call.

Open in Colab View on GitHub 6 code cells · 4 with saved output

Its imports and arguments are checked against shipit-agent 3.0. The outputs were saved when it was written, for an earlier release.

In [1]

python

from pathlib import Path
import sys
ROOT = Path.cwd().resolve().parent if Path.cwd().name == 'notebooks' else Path.cwd().resolve()
if str(ROOT) not in sys.path:
    sys.path.insert(0, str(ROOT))

from shipit_agent import Agent
from shipit_agent.computer_use import (
    BrowserAgentTool, MockBrowserSession,
)

Example 1 — Plain Agent + browser tool

We build a BrowserAgentTool against a MockBrowserSession (so the notebook runs offline). In production, swap the factory for PlaywrightBrowserSession.launch(...).

In [3]

python


class ScriptedLLM:
    def __init__(self, replies):
        self.replies = replies
        self._i = 0
    def complete(self, *, messages, **_):
        from shipit_agent.llms.base import LLMResponse
        text = self.replies[min(self._i, len(self.replies) - 1)]
        self._i += 1
        return LLMResponse(content=text)

class StringLLM:
    """For the sub-agent — accepts string complete() returns."""
    def __init__(self, replies):
        self.replies = replies
        self._i = 0
    def complete(self, *, messages, **_):
        text = self.replies[min(self._i, len(self.replies) - 1)]
        self._i += 1
        return text

In [4]

python

# The browser tool's sub-agent uses a vision-capable LLM (StringLLM here)
browser_tool = BrowserAgentTool(
    llm=StringLLM(replies=[
        'ACTION: navigate https://apple.com',
        'ACTION: click 240,80',
        'ACTION: done The iPhone 15 Pro starts at $999.',
    ]),
    browser_factory=lambda: MockBrowserSession(),
    max_iterations=8,
)

# Direct invocation of the tool — no main agent yet
result = browser_tool.run(goal='Find the iPhone 15 Pro starting price.')
print(result.text)
print()
print('metadata:')
for k in ('status', 'iterations'):
    print(f'  {k}: {result.metadata[k]}')

Out

Browser sub-agent finished (status=done, iterations=3).
Goal: Find the iPhone 15 Pro starting price.

Answer: The iPhone 15 Pro starts at $999.

Last actions:
  · iter 0: navigate
  · iter 1: click
  · iter 2: done

metadata:
  status: done
  iterations: 3

Example 2 — Main Agent calls `browser_use` as one tool among many

The main agent decides when to use the browser vs simpler tools (web_search, pdf_extract, etc.). The BrowserAgentTool shows up alongside them — same Tool protocol.

In [5]

python

browser_tool_2 = BrowserAgentTool(
    llm=StringLLM(replies=[
        'ACTION: navigate https://example.com/contact',
        'ACTION: type "Inquiry from research bot"',
        'ACTION: done Form filled and submitted.',
    ]),
    browser_factory=lambda: MockBrowserSession(),
    max_iterations=6,
)

# Toolset visible to the main agent
agent = Agent(
    llm=ScriptedLLM(replies=['I will respond directly.']),
    tools=[browser_tool_2],
    auto_use_skills=False,
)

print('Tools the main agent sees:')
for t in agent._effective_tools('any'):
    print(f'  - {t.name}: {t.description[:80]}...')

print()
print('Tool schema (what the model sees):')
import json
print(json.dumps(agent._effective_tools('any')[0].schema(), indent=2))

Out

Tools the main agent sees:
  - browser_use: Drive a real browser to accomplish a goal that requires reading and interacting ...

Tool schema (what the model sees):
{
  "type": "object",
  "properties": {
    "goal": {
      "type": "string",
      "description": "What the browser sub-agent should accomplish, in plain English. Be specific about what to extract or do."
    }
  },
  "required": [
    "goal"
  ]
}

Example 3 — `share_browser=True` for multi-step browser sessions

Trade-off: faster (no relaunch) but state leaks between calls — only enable when the parent agent is aware that browser state persists.

In [6]

python

browsers_made = []
def factory():
    b = MockBrowserSession()
    browsers_made.append(b)
    return b

shared_tool = BrowserAgentTool(
    llm=StringLLM(replies=[
        'ACTION: done step 1 finished',
        'ACTION: done step 2 finished',
        'ACTION: done step 3 finished',
    ]),
    browser_factory=factory,
    share_browser=True,   # ← reuse the browser across calls
)

for goal in [
    'Log in to the dashboard',
    'Open the billing page',
    'Download the latest invoice',
]:
    out = shared_tool.run(goal=goal)
    print(f'  → {out.text.splitlines()[0]}')

print()
print(f'browsers created: {len(browsers_made)}  (would be 3 without share_browser)')

# Always close at the end
shared_tool.close()
print('shared browser closed')

Out

  → Browser sub-agent finished (status=done, iterations=1).
  → Browser sub-agent finished (status=done, iterations=1).
  → Browser sub-agent finished (status=done, iterations=1).

browsers created: 1  (would be 3 without share_browser)
shared browser closed

Production setup

The same code, but with Playwright + a real LLM:

# pip install playwright
# playwright install chromium

from shipit_agent import Agent, VerifierNetwork
from shipit_agent.computer_use import (
    BrowserAgentTool, PlaywrightBrowserSession,
)

browser_tool = BrowserAgentTool(
    llm=opus_llm,                                  # vision-capable
    browser_factory=lambda: PlaywrightBrowserSession.launch(headless=True),
    max_iterations=12,
)

verifier = VerifierNetwork(llm=haiku_llm, goal='Research only — no purchases.')

agent = Agent(
    llm=opus_llm,                                  # main planner
    tools=[browser_tool, WebSearchTool(), PDFTool()],
    verifier=verifier,                             # gate destructive actions
)

result = agent.run(
    'Find the cheapest direct SFO-JFK flight on May 20 '
    'and summarise the booking page.'
)

Why this is powerful

Composable — browser_use is just a Tool. Add it to any Agent. Stack with RAG, structured output, verifier, memory consolidation.
Gated — pair with VerifierNetwork and the verifier vetoes destructive browser actions before they run.
Observable — every browser action lands in result.metadata['actions'], which flows through the parent agent's events.
Self-hosted — no Devin/Operator subscription. Your LLM, your infra, your data.

→ See Agent → ComputerUseAgent for the full reference.

from pathlib import Path import sys

ROOT = ( Path.cwd().resolve().parent if Path.cwd().name == "notebooks" else Path.cwd().resolve() ) if str(ROOT) not in sys.path: sys.path.insert(0, str(ROOT))

from examples.run_multi_tool_agent import build_llm_from_env from shipit_agent import Agent

!playwright install chromium

In [7]

python

!playwright install chromium

Out

Removing unused browser at ~/Library/Caches/ms-playwright/chromium-1223
Removing unused browser at ~/Library/Caches/ms-playwright/chromium_headless_shell-1223
Downloading Chromium 133.0.6943.16 (playwright build v1155) from https://cdn.playwright.dev/dbazure/download/playwright/builds/chromium/1155/chromium-mac-arm64.zip
123.3 MiB [                    ] 0% 0.0s123.3 MiB [                    ] 0% 100.8s123.3 MiB [                    ] 0% 58.0s123.3 MiB [                    ] 0% 66.7s123.3 MiB [                    ] 0% 50.5s123.3 MiB [                    ] 0% 50.3s123.3 MiB [                    ] 0% 28.9s123.3 MiB [                    ] 0% 30.0s123.3 MiB [                    ] 0% 22.5s123.3 MiB [                    ] 1% 14.4s123.3 MiB [                    ] 1% 13.2s123.3 MiB [                    ] 2% 10.0s123.3 MiB [                    ] 2% 11.0s123.3 MiB [=                   ] 2% 9.2s123.3 MiB [=               … (3,454 more characters)
Chromium 133.0.6943.16 (playwright build v1155) downloaded to ~/Library/Caches/ms-playwright/chromium-1155
Downloading Chromium Headless Shell 133.0.6943.16 (playwright build v1155) from https://cdn.playwright.dev/dbazure/download/playwright/builds/chromium/1155/chromium-headless-shell-mac-arm64.zip
77.9 MiB [                    ] 0% 0.0s77.9 MiB [                    ] 0% 65.5s77.9 MiB [                    ] 0% 43.8s77.9 MiB [                    ] 0% 28.8s77.9 MiB [                    ] 0% 18.0s77.9 MiB [                    ] 1% 12.0s77.9 MiB [                    ] 1% 11.1s77.9 MiB [                    ] 2% 8.9s77.9 MiB [=                   ] 2% 6.6s77.9 MiB [=                   ] 3% 5.6s77.9 MiB [=                   ] 4% 4.9s77.9 MiB [=                   ] 7% 3.3s77.9 MiB [==                  ] 9% 2.7s77.9 MiB [==                  ] 12% 2.2s77.9 MiB [===                 ] 14% 1.9s77.9 Mi… (1,594 more characters)
Chromium Headless Shell 133.0.6943.16 (playwright build v1155) downloaded to ~/Library/Caches/ms-playwright/chromium_headless_shell-1155

BrowserAgentTool — ComputerUseAgent inside the main Agent (v1.0.8)

Example 1 — Plain Agent + browser tool

Example 2 — Main Agent calls browser_use as one tool among many

Example 3 — share_browser=True for multi-step browser sessions

Production setup

Why this is powerful

BrowserAgentTool — ComputerUseAgent inside the main Agent (v1.0.8)

Example 1 — Plain Agent + browser tool

Example 2 — Main Agent calls browser_use as one tool among many

Example 3 — share_browser=True for multi-step browser sessions

Production setup

Why this is powerful

Example 2 — Main Agent calls `browser_use` as one tool among many

Example 3 — `share_browser=True` for multi-step browser sessions

Example 2 — Main Agent calls `browser_use` as one tool among many

Example 3 — `share_browser=True` for multi-step browser sessions