Notebooks · Reliability and control · 55

Verifier Network — Process Supervision (v1.0.8)

Name: SHIPIT Agent
Author: SHIPIT

Catch hallucinated tool calls and stalled agents before they waste tokens.

Open in Colab View on GitHub 5 code cells · 0 with saved output

Its imports and arguments are checked against shipit-agent 3.0. The outputs were saved when it was written, for an earlier release.

In [ ]

python

from pathlib import Path
import sys

ROOT = Path.cwd().resolve().parent if Path.cwd().name == 'notebooks' else Path.cwd().resolve()
if str(ROOT) not in sys.path:
    sys.path.insert(0, str(ROOT))

from shipit_agent import Agent
from shipit_agent.verifier import VerifierNetwork, VerifierConfig

1. Set up a verifier

The verifier needs its own LLM (cheap is fine — we recommend Haiku or gpt-oss-20b). Below uses a tiny scripted LLM so the notebook runs offline. Swap it for a real one (build_llm_from_env('haiku')) and the rest is identical.

In [ ]

python

import json

class CannedVerifierLLM:
    def __init__(self, replies):
        self.replies = replies
        self._i = 0
    def complete(self, *, messages, **_):
        out = self.replies[self._i]
        self._i += 1
        return out

verifier_llm = CannedVerifierLLM([
    json.dumps({'verdict': 'veto', 'reason': 'destructive', 'confidence': 0.95}),
    json.dumps({'verdict': 'allow', 'reason': 'safe read', 'confidence': 0.92}),
])

verifier = VerifierNetwork(
    llm=verifier_llm,
    config=VerifierConfig(veto_enabled=True, progress_enabled=True),
    goal='Read project files and produce a summary',
)

2. Pre-tool veto in action

Wrap any tool with the verifier. The wrapper is transparent — it has the same name, description, schema(), and run() surface as the wrapped tool.

In [ ]

python

class FakeTool:
    def __init__(self, name): self.name = name
    description = 'demo tool'
    prompt = ''
    prompt_instructions = ''
    def schema(self): return {'type': 'object'}
    def run(self, context=None, **kwargs):
        return {'text': f'{self.name} ran with {kwargs}'}

rm = FakeTool('rm_rf')
ls = FakeTool('list_files')
wrapped = verifier.wrap_tools([rm, ls])

# First call — verifier vetoes the destructive one
rm_result = wrapped[0].run(path='/')
print('rm result:', rm_result)

# Second call — verifier allows the safe one
ls_result = wrapped[1].run(path='./src')
print('ls result:', ls_result)

print()
print('telemetry:', verifier.stats)

3. Wire into `Agent` directly

Agent(verifier=...) automatically wraps every tool the agent uses. No other code changes needed — the runtime sees normal tools with normal schemas, but each one runs through the verifier before executing.

In [ ]

python

verifier2 = VerifierNetwork(
    llm=CannedVerifierLLM([json.dumps({'verdict': 'allow', 'confidence': 0.9})] * 4),
    goal='Inspect codebase',
)

# Faux LLM for the main agent that just echoes; in real life this is your model
class EchoLLM:
    def complete(self, *, messages, **_):
        from shipit_agent.llms.base import LLMResponse
        return LLMResponse(content='ok')

agent = Agent(
    llm=EchoLLM(),
    tools=[FakeTool('list_files'), FakeTool('read_file')],
    verifier=verifier2,
    auto_use_skills=False,
)

effective = agent._effective_tools('inspect')
print('Effective tools (verifier-wrapped):')
for t in effective:
    print(f'  {t.name} → {type(t).__name__}')

4. Progress check loop

After each agent iteration, score progress. When the agent stalls for progress_window iterations in a row, maybe_nudge() returns a user-message you can append to the conversation.

In [ ]

python

progress_llm = CannedVerifierLLM([
    json.dumps({'score': 0.2, 'summary': 'spinning'}),
    json.dumps({'score': 0.1, 'summary': 'still spinning'}),
    json.dumps({'score': 0.1, 'summary': 'stuck',
                'suggested_action': 'Try grep instead of read_file'}),
])
v3 = VerifierNetwork(
    llm=progress_llm,
    config=VerifierConfig(progress_window=3, progress_threshold=0.4),
    goal='Find the bug',
)

for i in range(3):
    last = v3.evaluate_step(last_step_summary=f'iteration {i+1}')
    print(f'iter {i+1} score: {last.score} streak: {v3.progress.streak_below}')

nudge = v3.maybe_nudge(last)
print()
print('nudge:', nudge)

5. Tuning

All thresholds live on VerifierConfig:

veto_min_confidence (0.0-1.0) — verdicts below this confidence get downgraded to ALLOW. Avoids over-blocking on uncertain calls.
progress_window (int) — consecutive sub-threshold iterations before nudging.
progress_threshold (0.0-1.0) — score below this counts as no-progress.
max_pretool_calls_per_run (int) — hard cap so the verifier itself can't cost you more than the main agent.
max_progress_calls_per_run (int) — same idea for the progress check.

Defaults are sensible for production agents (veto_min_confidence=0.6, progress_window=3, progress_threshold=0.4). You usually only adjust veto_min_confidence upward when you see the verifier blocking work it shouldn't, or progress_window downward when you want faster nudges.

Verifier Network — Process Supervision (v1.0.8)

Catch hallucinated tool calls and stalled agents before they waste tokens.

Its imports and arguments are checked against shipit-agent 3.0. The outputs were saved when it was written, for an earlier release.

In [ ]

python

from pathlib import Path
import sys

ROOT = Path.cwd().resolve().parent if Path.cwd().name == 'notebooks' else Path.cwd().resolve()
if str(ROOT) not in sys.path:
    sys.path.insert(0, str(ROOT))

from shipit_agent import Agent
from shipit_agent.verifier import VerifierNetwork, VerifierConfig

1. Set up a verifier

In [ ]

python

import json

class CannedVerifierLLM:
    def __init__(self, replies):
        self.replies = replies
        self._i = 0
    def complete(self, *, messages, **_):
        out = self.replies[self._i]
        self._i += 1
        return out

verifier_llm = CannedVerifierLLM([
    json.dumps({'verdict': 'veto', 'reason': 'destructive', 'confidence': 0.95}),
    json.dumps({'verdict': 'allow', 'reason': 'safe read', 'confidence': 0.92}),
])

verifier = VerifierNetwork(
    llm=verifier_llm,
    config=VerifierConfig(veto_enabled=True, progress_enabled=True),
    goal='Read project files and produce a summary',
)

2. Pre-tool veto in action

Wrap any tool with the verifier. The wrapper is transparent — it has the same name, description, schema(), and run() surface as the wrapped tool.

In [ ]

python

class FakeTool:
    def __init__(self, name): self.name = name
    description = 'demo tool'
    prompt = ''
    prompt_instructions = ''
    def schema(self): return {'type': 'object'}
    def run(self, context=None, **kwargs):
        return {'text': f'{self.name} ran with {kwargs}'}

rm = FakeTool('rm_rf')
ls = FakeTool('list_files')
wrapped = verifier.wrap_tools([rm, ls])

# First call — verifier vetoes the destructive one
rm_result = wrapped[0].run(path='/')
print('rm result:', rm_result)

# Second call — verifier allows the safe one
ls_result = wrapped[1].run(path='./src')
print('ls result:', ls_result)

print()
print('telemetry:', verifier.stats)

3. Wire into `Agent` directly

In [ ]

python

verifier2 = VerifierNetwork(
    llm=CannedVerifierLLM([json.dumps({'verdict': 'allow', 'confidence': 0.9})] * 4),
    goal='Inspect codebase',
)

# Faux LLM for the main agent that just echoes; in real life this is your model
class EchoLLM:
    def complete(self, *, messages, **_):
        from shipit_agent.llms.base import LLMResponse
        return LLMResponse(content='ok')

agent = Agent(
    llm=EchoLLM(),
    tools=[FakeTool('list_files'), FakeTool('read_file')],
    verifier=verifier2,
    auto_use_skills=False,
)

effective = agent._effective_tools('inspect')
print('Effective tools (verifier-wrapped):')
for t in effective:
    print(f'  {t.name} → {type(t).__name__}')

4. Progress check loop

After each agent iteration, score progress. When the agent stalls for progress_window iterations in a row, maybe_nudge() returns a user-message you can append to the conversation.

In [ ]

python

progress_llm = CannedVerifierLLM([
    json.dumps({'score': 0.2, 'summary': 'spinning'}),
    json.dumps({'score': 0.1, 'summary': 'still spinning'}),
    json.dumps({'score': 0.1, 'summary': 'stuck',
                'suggested_action': 'Try grep instead of read_file'}),
])
v3 = VerifierNetwork(
    llm=progress_llm,
    config=VerifierConfig(progress_window=3, progress_threshold=0.4),
    goal='Find the bug',
)

for i in range(3):
    last = v3.evaluate_step(last_step_summary=f'iteration {i+1}')
    print(f'iter {i+1} score: {last.score} streak: {v3.progress.streak_below}')

nudge = v3.maybe_nudge(last)
print()
print('nudge:', nudge)

5. Tuning

All thresholds live on VerifierConfig:

veto_min_confidence (0.0-1.0) — verdicts below this confidence get downgraded to ALLOW. Avoids over-blocking on uncertain calls.
progress_window (int) — consecutive sub-threshold iterations before nudging.
progress_threshold (0.0-1.0) — score below this counts as no-progress.
max_pretool_calls_per_run (int) — hard cap so the verifier itself can't cost you more than the main agent.
max_progress_calls_per_run (int) — same idea for the progress check.

Verifier Network — Process Supervision (v1.0.8)

1. Set up a verifier

2. Pre-tool veto in action

3. Wire into Agent directly

4. Progress check loop

5. Tuning

Verifier Network — Process Supervision (v1.0.8)

1. Set up a verifier

2. Pre-tool veto in action

3. Wire into Agent directly

4. Progress check loop

5. Tuning

3. Wire into `Agent` directly

3. Wire into `Agent` directly