Notebooks · In-depth guides · Guide 12

Cost, tracing, serving and output

Name: SHIPIT Agent
Author: SHIPIT

Running agents for real means knowing what each run cost, keeping a record of what happened, showing it to people, and sending it over the web. This guide covers all four.

Open in Colab View on GitHub 13 code cells · 11 with saved output

Run end to end on shipit-agent 3.0 with real model calls (Haiku 5.5 on Anthropic, Gemma 4 31B on Bedrock). Every output below is from that run.

1 · Models and a small tool

In [1]

python

import json
import os
import shutil
from pathlib import Path

from shipit_agent import Agent
from shipit_agent.llms import build_llm_from_env
from shipit_agent.llms.factory import build_llm_from_settings
from shipit_agent.tools import FunctionTool

os.environ["SHIPIT_ANTHROPIC_MODEL"] = "claude-haiku-5-5"
haiku = build_llm_from_env("anthropic")
gemma = build_llm_from_settings({"model": "google.gemma-4-31b"}, provider="bedrock")

STOCK = {"widget": 42, "gadget": 0, "gizmo": 7}

def stock(item: str) -> str:
    """Units in stock for an item (widget, gadget or gizmo)."""
    return str(STOCK.get(item.lower(), "unknown item"))

stock_tool = FunctionTool.from_callable(stock, read_only=True)
WORK = Path("ops_demo")
shutil.rmtree(WORK, ignore_errors=True)

2 · Cost per run

A CostTracker attached as hooks records every model call and prices it from shipit's pricing table, including prompt-cache reads and writes.

In [2]

python

from shipit_agent.costs import Budget, CostTracker

tracker = CostTracker()
Agent(llm=haiku, tools=[stock_tool], hooks=tracker.as_hooks()).run("Which items are out of stock? Check each.")
Agent(llm=gemma, tools=[stock_tool], hooks=tracker.as_hooks()).run("How many gizmos are left?")

summary = tracker.summary()
print(f"total: ${summary['total_cost_usd']:.5f} over {summary['total_calls']} model calls")
print("tokens:", summary["total_tokens"])

Out

total: $0.00042 over 4 model calls
tokens: {'input_tokens': 1888, 'output_tokens': 260, 'cache_read_tokens': 3294, 'cache_write_tokens': 0}

3 · Per-call breakdown

In [3]

python

for row in tracker.breakdown():
    print(f"{row['model']:22} in {row['input_tokens']:6} out {row['output_tokens']:5} "
          f"cache-read {row['cache_read_tokens']:6}  ${row['cost_usd']:.5f}")

Out

claude-haiku-5-5       in      4 out   189 cache-read   1622  $0.00011
claude-haiku-5-5       in      2 out    39 cache-read   1672  $0.00004
google.gemma-4-31b     in    973 out    23 cache-read      0  $0.00015
google.gemma-4-31b     in    909 out     9 cache-read      0  $0.00013

4 · A budget that warns, then stops

Budget(max_dollars, warn_at) calls your alert at the warning threshold and raises once the limit is passed. A tiny limit shows both.

In [4]

python

alerts = []
tight = CostTracker(budget=Budget(max_dollars=0.0003, warn_at=0.5),
                    on_cost_alert=lambda spent, limit: alerts.append(f"${spent:.5f} of ${limit:.5f}"))
agent = Agent(llm=haiku, tools=[stock_tool], hooks=tight.as_hooks())
try:
    for item in ("widget", "gadget", "gizmo"):
        agent.run(f"How many {item}s are in stock?")
        print(f"after {item}: ${tight.total_cost:.5f}")
except Exception as error:
    print("stopped:", type(error).__name__, "-", str(error)[:100])
print("alerts:", alerts)

Out

after widget: $0.00006

Cost alert: $0.0002 spent of $0.00 budget (58%)

after gadget: $0.00017

stopped: BudgetExceededError - Budget exceeded: $0.0003 spent of $0.0003 limit (model: claude-haiku-5-5)
alerts: ['$0.00017 of $0.00030']

max_budget_usd= on the agent is the simpler form: the run stops between steps once its own estimated cost reaches the limit.

5 · Record a run to a trace

A trace_store saves every event of a run under its trace_id.

In [5]

python

from shipit_agent import FileTraceStore

traces = FileTraceStore(root_dir=WORK / "traces")
Agent(llm=haiku, tools=[stock_tool], trace_store=traces, trace_id="stock-haiku").run("Is any widget left?")
Agent(llm=gemma, tools=[stock_tool], trace_store=traces, trace_id="stock-gemma").run("Is any widget left?")
print(sorted(p.name for p in (WORK / "traces").iterdir()))

Out

['stock-gemma.json', 'stock-haiku.json']

6 · Load a trace back

A TraceRecord holds the trace id, metadata and every event, in order.

In [6]

python

from collections import Counter

record = traces.load("stock-haiku")
print(record.trace_id, "|", len(record.events), "events")
print(dict(Counter(e.type for e in record.events)))
final = next(e for e in reversed(record.events) if e.type == "run_completed")
print("answer:", final.payload["output"])

Out

stock-haiku | 21 events
{'run_started': 1, 'step_started': 2, 'usage_tick': 2, 'assistant_message': 2, 'tool_group_started': 1, 'tool_called': 1, 'tool_output_started': 1, 'tool_output_delta': 1, 'tool_completed': 1, 'tool_group_completed': 1, 'text_delta': 5, 'final_answer': 1, 'run_summary': 1, 'run_completed': 1}
answer: Yes. There are 42 widgets in stock.

7 · Compare two runs

diff_traces lines up two runs, here the same question on two models.

In [7]

python

from shipit_agent import TraceReplayer, diff_traces

diff = diff_traces(TraceReplayer.from_store(traces, "stock-haiku"), TraceReplayer.from_store(traces, "stock-gemma"))
print("same events before they diverge:", diff.matched, "| diverged at event", diff.diverged_at)
print("only in the Haiku run:", Counter(e.type for _, e in diff.only_in_left))
print("only in the Gemma run:", Counter(e.type for _, e in diff.only_in_right))

Out

same events before they diverge: 16 | diverged at event 16
only in the Haiku run: Counter()
only in the Gemma run: Counter({'usage_tick': 1, 'assistant_message': 1, 'final_answer': 1, 'run_summary': 1, 'run_completed': 1})

8 · A transcript as text

render_transcript turns events (a list, a trace, or a result) into the compact transcript a terminal shows.

In [8]

python

from shipit_agent import render_transcript

result = Agent(llm=haiku, tools=[stock_tool]).run("Check widget and gizmo stock, then say which to reorder first.")
print(render_transcript(result.events, model="claude-haiku-5-5", width=100))

Out

  ◆ Ran stock 2 times ›
    widget · gizmo

**Reorder the gizmo first.** It has 7 units in stock versus 42 for the widget.

The tool returns only current counts, with no reorder points or sales rates, so this is based on stock levels alone. If the widget sells much faster than the gizmo, the ranking could change.

                                                 250 tokens · claude-haiku-5-5

9 · A transcript as a tree

In [9]

python

from shipit_agent import render_tree

print(render_tree(result.events, model="claude-haiku-5-5"))

Out

Agent started
│
├─ Tool group: Ran stock 2 times
│  ├─ stock                                         completed  0ms
│  └─ stock                                         completed  0ms
│
└─ Final answer
   **Reorder the gizmo first.** It has 7 units in stock versus 42 for the widget.
   
   The tool returns only current counts, with no reorder points or sales rates, so this is based
   on stock levels alone. If the widget sells much faster than the gizmo, the ranking could
   change.

250 tokens · claude-haiku-5-5

10 · A shareable HTML page

render_transcript_html writes a self-contained page of the run: the prompt, each step, the tool calls and the answer. It renders here as live HTML.

In [10]

python

from IPython.display import HTML

from shipit_agent import render_transcript_html

page = render_transcript_html(result.events, title="Stock check", model="claude-haiku-5-5",
                              prompt="Check widget and gizmo stock, then say which to reorder first.")
(WORK / "stock-check.html").write_text(page)
print(len(page), "characters")
HTML(page)

Out

4065 characters

rendered output

11 · Packets for a web client

event_packet and result_packet turn events and results into plain JSON dicts with a stable shape, for a browser or another service.

In [11]

python

from shipit_agent import event_packet, result_packet

called = next(e for e in result.events if e.type == "tool_called")
print(json.dumps(event_packet(called), default=str)[:400])
print(sorted(result_packet(result)))

Out

{"packet_type": "agent_event", "event": {"type": "tool_called", "message": "Tool called: stock", "payload": {"tool": "stock", "call_id": "toolu_019eAwPCgCPwrYvFUPiTLaq5", "group_id": "tool_group_1", "arguments": {"item": "widget"}, "iteration": 1, "run_id": "08244d35-4d1a-4017-9e33-f0e9691b984e", "session_id": "94a4f5f5-f7ad-4913-8705-f65e93358321", "sequence": 6, "tool_call_id": "toolu_019eAwPCgC
['packet_type', 'result']

12 · Server-Sent Events

agent.stream_sse(prompt) yields ready-to-send SSE frames. A web framework returns them as a streaming response with content-type: text/event-stream.

In [12]

python

frames = list(Agent(llm=haiku, tools=[stock_tool]).stream_sse("How many gizmos?"))
print(len(frames), "frames")
for frame in frames[:3] + frames[-1:]:
    print(repr(frame[:150]))

Out

21 frames
'id: 0\nevent: stream_hello\ndata: {"type": "stream_hello", "durability": "control", "generation": "89657-1794143791304", "sequence": 0, "message": "Stre'
'id: 1\nevent: run_started\ndata: {"type": "run_started", "durability": "canonical", "generation": "89657-1794143791304", "sequence": 1, "message": "Agen'
'id: 2\nevent: step_started\ndata: {"type": "step_started", "durability": "provisional", "generation": "89657-1794143791304", "sequence": 2, "message": "'
'event: done\ndata: [DONE]\n\n'

In FastAPI, for example:

from fastapi.responses import StreamingResponse

@app.post("/ask")
def ask(body: Question):
    agent = Agent(llm=llm, tools=[stock_tool])     # one agent per request: one conversation per caller
    return StreamingResponse(agent.stream_sse(body.text), media_type="text/event-stream")

shipit serve runs a ready-made server; see the deployment guide.

13 · Clean up

In [13]

python

shutil.rmtree(WORK, ignore_errors=True)

14 · Summary

CostTracker prices every call (with cache tokens), breaks it down, and enforces a Budget.
trace_store + trace_id record runs; TraceReplayer and diff_traces load and compare them.
render_transcript, render_tree and render_transcript_html show a run to people.
event_packet, result_packet and stream_sse send it over the web.

Next: Guide 13 — chat history.

Notebooks · In-depth guides · Guide 12

Cost, tracing, serving and output

Running agents for real means knowing what each run cost, keeping a record of what happened, showing it to people, and sending it over the web. This guide covers all four.

Open in Colab View on GitHub 13 code cells · 11 with saved output

Run end to end on shipit-agent 3.0 with real model calls (Haiku 5.5 on Anthropic, Gemma 4 31B on Bedrock). Every output below is from that run.

1 · Models and a small tool

In [1]

python

import json
import os
import shutil
from pathlib import Path

from shipit_agent import Agent
from shipit_agent.llms import build_llm_from_env
from shipit_agent.llms.factory import build_llm_from_settings
from shipit_agent.tools import FunctionTool

os.environ["SHIPIT_ANTHROPIC_MODEL"] = "claude-haiku-5-5"
haiku = build_llm_from_env("anthropic")
gemma = build_llm_from_settings({"model": "google.gemma-4-31b"}, provider="bedrock")

STOCK = {"widget": 42, "gadget": 0, "gizmo": 7}

def stock(item: str) -> str:
    """Units in stock for an item (widget, gadget or gizmo)."""
    return str(STOCK.get(item.lower(), "unknown item"))

stock_tool = FunctionTool.from_callable(stock, read_only=True)
WORK = Path("ops_demo")
shutil.rmtree(WORK, ignore_errors=True)

2 · Cost per run

A CostTracker attached as hooks records every model call and prices it from shipit's pricing table, including prompt-cache reads and writes.

In [2]

python

from shipit_agent.costs import Budget, CostTracker

tracker = CostTracker()
Agent(llm=haiku, tools=[stock_tool], hooks=tracker.as_hooks()).run("Which items are out of stock? Check each.")
Agent(llm=gemma, tools=[stock_tool], hooks=tracker.as_hooks()).run("How many gizmos are left?")

summary = tracker.summary()
print(f"total: ${summary['total_cost_usd']:.5f} over {summary['total_calls']} model calls")
print("tokens:", summary["total_tokens"])

Out

total: $0.00042 over 4 model calls
tokens: {'input_tokens': 1888, 'output_tokens': 260, 'cache_read_tokens': 3294, 'cache_write_tokens': 0}

3 · Per-call breakdown

In [3]

python

for row in tracker.breakdown():
    print(f"{row['model']:22} in {row['input_tokens']:6} out {row['output_tokens']:5} "
          f"cache-read {row['cache_read_tokens']:6}  ${row['cost_usd']:.5f}")

Out

claude-haiku-5-5       in      4 out   189 cache-read   1622  $0.00011
claude-haiku-5-5       in      2 out    39 cache-read   1672  $0.00004
google.gemma-4-31b     in    973 out    23 cache-read      0  $0.00015
google.gemma-4-31b     in    909 out     9 cache-read      0  $0.00013

4 · A budget that warns, then stops

Budget(max_dollars, warn_at) calls your alert at the warning threshold and raises once the limit is passed. A tiny limit shows both.

In [4]

python

alerts = []
tight = CostTracker(budget=Budget(max_dollars=0.0003, warn_at=0.5),
                    on_cost_alert=lambda spent, limit: alerts.append(f"${spent:.5f} of ${limit:.5f}"))
agent = Agent(llm=haiku, tools=[stock_tool], hooks=tight.as_hooks())
try:
    for item in ("widget", "gadget", "gizmo"):
        agent.run(f"How many {item}s are in stock?")
        print(f"after {item}: ${tight.total_cost:.5f}")
except Exception as error:
    print("stopped:", type(error).__name__, "-", str(error)[:100])
print("alerts:", alerts)

Out

after widget: $0.00006

Cost alert: $0.0002 spent of $0.00 budget (58%)

after gadget: $0.00017

stopped: BudgetExceededError - Budget exceeded: $0.0003 spent of $0.0003 limit (model: claude-haiku-5-5)
alerts: ['$0.00017 of $0.00030']

max_budget_usd= on the agent is the simpler form: the run stops between steps once its own estimated cost reaches the limit.

5 · Record a run to a trace

A trace_store saves every event of a run under its trace_id.

In [5]

python

from shipit_agent import FileTraceStore

traces = FileTraceStore(root_dir=WORK / "traces")
Agent(llm=haiku, tools=[stock_tool], trace_store=traces, trace_id="stock-haiku").run("Is any widget left?")
Agent(llm=gemma, tools=[stock_tool], trace_store=traces, trace_id="stock-gemma").run("Is any widget left?")
print(sorted(p.name for p in (WORK / "traces").iterdir()))

Out

['stock-gemma.json', 'stock-haiku.json']

6 · Load a trace back

A TraceRecord holds the trace id, metadata and every event, in order.

In [6]

python

from collections import Counter

record = traces.load("stock-haiku")
print(record.trace_id, "|", len(record.events), "events")
print(dict(Counter(e.type for e in record.events)))
final = next(e for e in reversed(record.events) if e.type == "run_completed")
print("answer:", final.payload["output"])

Out

stock-haiku | 21 events
{'run_started': 1, 'step_started': 2, 'usage_tick': 2, 'assistant_message': 2, 'tool_group_started': 1, 'tool_called': 1, 'tool_output_started': 1, 'tool_output_delta': 1, 'tool_completed': 1, 'tool_group_completed': 1, 'text_delta': 5, 'final_answer': 1, 'run_summary': 1, 'run_completed': 1}
answer: Yes. There are 42 widgets in stock.

7 · Compare two runs

diff_traces lines up two runs, here the same question on two models.

In [7]

python

from shipit_agent import TraceReplayer, diff_traces

diff = diff_traces(TraceReplayer.from_store(traces, "stock-haiku"), TraceReplayer.from_store(traces, "stock-gemma"))
print("same events before they diverge:", diff.matched, "| diverged at event", diff.diverged_at)
print("only in the Haiku run:", Counter(e.type for _, e in diff.only_in_left))
print("only in the Gemma run:", Counter(e.type for _, e in diff.only_in_right))

Out

same events before they diverge: 16 | diverged at event 16
only in the Haiku run: Counter()
only in the Gemma run: Counter({'usage_tick': 1, 'assistant_message': 1, 'final_answer': 1, 'run_summary': 1, 'run_completed': 1})

8 · A transcript as text

render_transcript turns events (a list, a trace, or a result) into the compact transcript a terminal shows.

In [8]

python

from shipit_agent import render_transcript

result = Agent(llm=haiku, tools=[stock_tool]).run("Check widget and gizmo stock, then say which to reorder first.")
print(render_transcript(result.events, model="claude-haiku-5-5", width=100))

Out

  ◆ Ran stock 2 times ›
    widget · gizmo

**Reorder the gizmo first.** It has 7 units in stock versus 42 for the widget.

The tool returns only current counts, with no reorder points or sales rates, so this is based on stock levels alone. If the widget sells much faster than the gizmo, the ranking could change.

                                                 250 tokens · claude-haiku-5-5

9 · A transcript as a tree

In [9]

python

from shipit_agent import render_tree

print(render_tree(result.events, model="claude-haiku-5-5"))

Out

Agent started
│
├─ Tool group: Ran stock 2 times
│  ├─ stock                                         completed  0ms
│  └─ stock                                         completed  0ms
│
└─ Final answer
   **Reorder the gizmo first.** It has 7 units in stock versus 42 for the widget.
   
   The tool returns only current counts, with no reorder points or sales rates, so this is based
   on stock levels alone. If the widget sells much faster than the gizmo, the ranking could
   change.

250 tokens · claude-haiku-5-5

10 · A shareable HTML page

render_transcript_html writes a self-contained page of the run: the prompt, each step, the tool calls and the answer. It renders here as live HTML.

In [10]

python

from IPython.display import HTML

from shipit_agent import render_transcript_html

page = render_transcript_html(result.events, title="Stock check", model="claude-haiku-5-5",
                              prompt="Check widget and gizmo stock, then say which to reorder first.")
(WORK / "stock-check.html").write_text(page)
print(len(page), "characters")
HTML(page)

Out

4065 characters

rendered output

11 · Packets for a web client

event_packet and result_packet turn events and results into plain JSON dicts with a stable shape, for a browser or another service.

In [11]

python

from shipit_agent import event_packet, result_packet

called = next(e for e in result.events if e.type == "tool_called")
print(json.dumps(event_packet(called), default=str)[:400])
print(sorted(result_packet(result)))

Out

{"packet_type": "agent_event", "event": {"type": "tool_called", "message": "Tool called: stock", "payload": {"tool": "stock", "call_id": "toolu_019eAwPCgCPwrYvFUPiTLaq5", "group_id": "tool_group_1", "arguments": {"item": "widget"}, "iteration": 1, "run_id": "08244d35-4d1a-4017-9e33-f0e9691b984e", "session_id": "94a4f5f5-f7ad-4913-8705-f65e93358321", "sequence": 6, "tool_call_id": "toolu_019eAwPCgC
['packet_type', 'result']

12 · Server-Sent Events

agent.stream_sse(prompt) yields ready-to-send SSE frames. A web framework returns them as a streaming response with content-type: text/event-stream.

In [12]

python

frames = list(Agent(llm=haiku, tools=[stock_tool]).stream_sse("How many gizmos?"))
print(len(frames), "frames")
for frame in frames[:3] + frames[-1:]:
    print(repr(frame[:150]))

Out

21 frames
'id: 0\nevent: stream_hello\ndata: {"type": "stream_hello", "durability": "control", "generation": "89657-1794143791304", "sequence": 0, "message": "Stre'
'id: 1\nevent: run_started\ndata: {"type": "run_started", "durability": "canonical", "generation": "89657-1794143791304", "sequence": 1, "message": "Agen'
'id: 2\nevent: step_started\ndata: {"type": "step_started", "durability": "provisional", "generation": "89657-1794143791304", "sequence": 2, "message": "'
'event: done\ndata: [DONE]\n\n'

In FastAPI, for example:

from fastapi.responses import StreamingResponse

@app.post("/ask")
def ask(body: Question):
    agent = Agent(llm=llm, tools=[stock_tool])     # one agent per request: one conversation per caller
    return StreamingResponse(agent.stream_sse(body.text), media_type="text/event-stream")

shipit serve runs a ready-made server; see the deployment guide.

13 · Clean up

In [13]

python

shutil.rmtree(WORK, ignore_errors=True)

14 · Summary

CostTracker prices every call (with cache tokens), breaks it down, and enforces a Budget.
trace_store + trace_id record runs; TraceReplayer and diff_traces load and compare them.
render_transcript, render_tree and render_transcript_html show a run to people.
event_packet, result_packet and stream_sse send it over the web.

Next: Guide 13 — chat history.