Loading the notebook…
Loading the notebook…
Notebooks · Provider API features · 65
Two v1.0.12 features in one notebook:
Its imports and arguments are checked against shipit-agent 3.0. The outputs were saved when it was written, for an earlier release.
document content blocks with
{"citations": {"enabled": True}}, parsed back from the response text blocks. They
work with the Anthropic API directly and with Anthropic models via Bedrock / LiteLLM.
They do not apply to OpenAI / Gemini / Groq / Ollama.BatchRuntime targets Anthropic.)from pathlib import Path
import sys
ROOT = (
Path.cwd().resolve().parent
if Path.cwd().name == "notebooks"
else Path.cwd().resolve()
)
if str(ROOT) not in sys.path:
sys.path.insert(0, str(ROOT))Each helper builds a document content block with citations enabled by default
(that's the whole point). The source shapes match the Anthropic SDK:
text_document(text) — plain text (source.type == "text")pdf_document(base64) — base64 PDF (source.type == "base64")url_pdf_document(url) — URL-sourced PDF (source.type == "url")content_document(content) — built from content blocks (source.type == "content")import json
from shipit_agent.llms import (
text_document,
pdf_document,
url_pdf_document,
content_document,
extract_citations,
)
doc = text_document(
"The SHIPIT Batch API bills at roughly 50% of standard per-token pricing.",
title="Pricing FAQ",
context="Internal pricing doc",
)
print(json.dumps(doc, indent=2)){
"type": "document",
"source": {
"type": "text",
"media_type": "text/plain",
"data": "The SHIPIT Batch API bills at roughly 50% of standard per-token pricing."
},
"title": "Pricing FAQ",
"context": "Internal pricing doc",
"citations": {
"enabled": true
}
}# Citations can be turned off per document, and PDF/URL variants have the same shape.
print("pdf (citations off):", json.dumps(pdf_document("BASE64DATA", citations=False)))
print("url pdf: ", json.dumps(url_pdf_document("https://example.com/report.pdf")))
print("content document: ", json.dumps(content_document("inline grounding text")))pdf (citations off): {"type": "document", "source": {"type": "base64", "media_type": "application/pdf", "data": "BASE64DATA"}}
url pdf: {"type": "document", "source": {"type": "url", "url": "https://example.com/report.pdf"}, "citations": {"enabled": true}}
content document: {"type": "document", "source": {"type": "content", "content": "inline grounding text"}, "citations": {"enabled": true}}Pass documents=[...] to complete() (or set them on the adapter). The adapter
prepends the document blocks to the last user message so the model can ground its
answer. We inspect that offline via _build_request_kwargs(...).
from shipit_agent.llms import AnthropicChatLLM
from shipit_agent.models import Message
llm = AnthropicChatLLM(model="claude-opus-4-1", api_key="offline-demo-key")
req = llm._build_request_kwargs(
messages=[Message(role="user", content="What discount does the Batch API give?")],
tools=None,
system_prompt=None,
documents=[text_document(
"The Batch API processes requests asynchronously at ~50% of standard price.",
title="Pricing FAQ",
)],
)
last_user = req["messages"][-1]
print("last user message content blocks (document prepended before the question):")
print(json.dumps(last_user["content"], indent=2))last user message content blocks (document prepended before the question):
[
{
"type": "document",
"source": {
"type": "text",
"media_type": "text/plain",
"data": "The Batch API processes requests asynchronously at ~50% of standard price."
},
"title": "Pricing FAQ",
"citations": {
"enabled": true
}
},
{
"type": "text",
"text": "What discount does the Batch API give?"
}
]extract_citations(content_blocks) walks the response text blocks, reads each block's
citations (location objects like char_location / page_location), and returns
them as plain dicts — exactly what the adapter stores in metadata['citations'].
It's standalone and defensive, so we can call it directly on fake response blocks.
from types import SimpleNamespace as NS
# A fake response: one cited text block + one uncited block.
fake_blocks = [
NS(type="text", text="The Batch API gives about a 50% discount", citations=[
{"type": "char_location", "document_index": 0, "document_title": "Pricing FAQ",
"start_char_index": 0, "end_char_index": 62,
"cited_text": "processes requests asynchronously at ~50% of standard price"},
]),
NS(type="text", text=" for latency-tolerant work.", citations=None),
]
cites = extract_citations(fake_blocks)
print("metadata['citations'] would be:")
print(json.dumps(cites, indent=2))
print("\nnon-citation responses simply yield:", extract_citations([NS(type="text", text="no cites")]))metadata['citations'] would be:
[
{
"type": "char_location",
"document_index": 0,
"document_title": "Pricing FAQ",
"start_char_index": 0,
"end_char_index": 62,
"cited_text": "processes requests asynchronously at ~50% of standard price"
}
]
non-citation responses simply yield: []In complete() the adapter calls extract_citations(response.content) and, when
non-empty, stores the result under metadata['citations'] alongside the answer text.
BatchRuntime wraps the Anthropic Messages Batches API. You build a list of
BatchRequests, call runtime.run(...), and get back BatchResults. Batches are
processed asynchronously (up to 24h) and billed at ~50% of the standard
per-token price — ideal for evals, backfills, nightly summarisation and labelling.
A BatchRequest needs a unique custom_id and either a prompt or explicit
messages. to_payload() shows the exact {custom_id, params} entry sent to the SDK.
from shipit_agent.batch import BatchRequest, BatchResult, BatchRuntime
requests = [
BatchRequest(custom_id="t1", prompt="Summarise: customer wants a refund.", system="Be terse."),
BatchRequest(custom_id="t2", prompt="Classify sentiment: I love this product!"),
BatchRequest(
custom_id="t3",
messages=[{"role": "user", "content": "Translate to French: good morning"}],
model="claude-3-5-haiku-latest",
max_tokens=128,
),
]
print(json.dumps(requests[0].to_payload(), indent=2)){
"custom_id": "t1",
"params": {
"model": "claude-3-5-sonnet-latest",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": "Summarise: customer wants a refund."
}
],
"system": "Be terse."
}
}BatchRuntime accepts an injected client (anything exposing
messages.batches.create/retrieve/results/cancel) and run(...) takes injectable
sleep/now callables. We mirror the test-suite's fake client so this executes with
no network and no real sleeping: the batch reports in_progress twice, then ended.
from types import SimpleNamespace as NS
def _text_block(text):
return NS(type="text", text=text)
def _succeeded(custom_id, text):
message = NS(
content=[_text_block(text)],
usage=NS(input_tokens=12, output_tokens=8),
stop_reason="end_turn",
)
return NS(custom_id=custom_id, result=NS(type="succeeded", message=message))
def _errored(custom_id, message):
error = NS(error=NS(message=message))
return NS(custom_id=custom_id, result=NS(type="errored", error=error))
class _FakeBatch:
def __init__(self, batch_id, status):
self.id = batch_id
self.processing_status = status
class _FakeBatches:
def __init__(self, statuses, results, batch_id="batch_demo"):
self._statuses = list(statuses)
self._results = results
self._batch_id = batch_id
self.created_with = None
self.retrieve_calls = 0
def create(self, *, requests):
self.created_with = requests
return _FakeBatch(self._batch_id, self._statuses[0])
def retrieve(self, batch_id):
self.retrieve_calls += 1
idx = min(self.retrieve_calls, len(self._statuses) - 1)
return _FakeBatch(batch_id, self._statuses[idx])
def results(self, batch_id):
return list(self._results)
class _FakeClient:
def __init__(self, batches):
self.messages = NS(batches=batches)
batches = _FakeBatches(
statuses=["in_progress", "in_progress", "ended"],
results=[
_succeeded("t1", "Customer requests a refund."),
_succeeded("t2", "positive"),
_errored("t3", "rate limited"),
],
)
runtime = BatchRuntime(_FakeClient(batches))
sleeps = []
results = runtime.run(requests, poll_interval=30.0, sleep=sleeps.append) # no-op sleep
print(f"polled {batches.retrieve_calls} times; recorded sleeps: {sleeps}")
for r in results:
status = "OK" if r.ok else f"ERROR: {r.error}"
print(f" {r.custom_id}: {status:<22} output={r.output!r}")polled 2 times; recorded sleeps: [30.0] t1: OK output='Customer requests a refund.' t2: OK output='positive' t3: ERROR: rate limited output=''
BatchResult.ok is True only on a succeeded result with no error; errored,
canceled and expired results capture a human-readable error instead of raising.
The raw SDK result is kept on .raw for callers that need stop reasons / request ids.
You can also drive the lifecycle by hand: submit(...) -> batch_id, status(id),
results(id), and cancel(id).
batch_id = runtime.submit(requests)
print("submitted batch id:", batch_id)
print("payload sent to create():")
print(json.dumps(batches.created_with[0], indent=2))submitted batch id: batch_demo
payload sent to create():
{
"custom_id": "t1",
"params": {
"model": "claude-3-5-sonnet-latest",
"max_tokens": 1024,
"messages": [
{
"role": "user",
"content": "Summarise: customer wants a refund."
}
],
"system": "Be terse."
}
}This cell submits a real batch to Anthropic. It only runs when ANTHROPIC_API_KEY
is set. Real batches can take up to 24h, so we just submit and print the id rather
than polling to completion.
import os
if os.getenv("ANTHROPIC_API_KEY"):
live = BatchRuntime(api_key=os.environ["ANTHROPIC_API_KEY"])
bid = live.submit([
BatchRequest(custom_id="demo", prompt="Say hello in one word.", max_tokens=16),
])
print("submitted real batch:", bid, "status:", live.status(bid))
print("(batches finish asynchronously; fetch with runtime.results(bid) once ended)")
else:
print("Set ANTHROPIC_API_KEY to submit a real batch. Skipping (offline).")Set ANTHROPIC_API_KEY to submit a real batch. Skipping (offline).
text_document / pdf_document / url_pdf_document /
content_document build cited document blocks; the adapter attaches them to the
last user message and extract_citations(...) parses grounded citations into
metadata['citations'].BatchRuntime.run([...]) submits, polls until ended, and maps to
BatchResults — billed at ~50% of standard pricing. Inject a fake client +
no-op sleep to run it entirely offline.