Notebooks · Autopilot · 39

Persistence, resume, and the scheduler daemon

Name: SHIPIT Agent
Author: SHIPIT

Autopilot checkpoints every iteration so a crash is cheap. The SchedulerDaemon takes that further — a persistent goal queue on disk, drained tick-by-tick until SIGINT. This is the piece that lets shipit_agent run for…

Open in Colab View on GitHub 6 code cells · 0 with saved output

Its imports and arguments are checked against shipit-agent 3.0. The outputs were saved when it was written, for an earlier release.

In [ ]

python

from pathlib import Path
import sys

# Resolve the repo root whether this cell is running from ./notebooks
# or from the repo root — mirrors the existing 01–36 notebook series.
ROOT = (
    Path.cwd().resolve().parent
    if Path.cwd().name == "notebooks"
    else Path.cwd().resolve()
)
if str(ROOT) not in sys.path:
    sys.path.insert(0, str(ROOT))

from examples.run_multi_tool_agent import build_llm_from_env

# Bedrock Llama 4 Scout is the default model this library ships with
# (see `shipit_agent/config.py`). No model name required — the helper
# reads `AWS_REGION` / credentials from env and returns an adapter that
# works with every Autopilot / Agent class.
llm = build_llm_from_env("bedrock")
print("Bedrock LLM ready:", type(llm).__name__)

1. Crash → resume round-trip

We'll simulate a crash by capping max_iterations=1 for the first run, then resume with a larger cap.

In [ ]

python

from shipit_agent.autopilot import Autopilot, BudgetPolicy
from shipit_agent.deep import Goal

goal = Goal(
    objective="Outline a simple CLI todo app architecture.",
    success_criteria=[
        "Files listed",
        "CLI commands described",
        "Storage choice justified",
    ],
)

# First run — tiny budget, expected to halt partial.
first = Autopilot(
    llm=llm,
    goal=goal,
    budget=BudgetPolicy(max_iterations=1, max_seconds=120),
).run(run_id="todo-arch")

print(f"First run → {first.status} (iters={first.iterations})")
print(f"Halt: {first.halt_reason}")

Now resume — Autopilot loads the checkpoint and keeps going.

In [ ]

python

second = Autopilot(
    llm=llm,
    goal=goal,
    budget=BudgetPolicy(max_iterations=6, max_seconds=240),
).resume("todo-arch")

print(f"Resumed → {second.status} (iters={second.iterations})")
print(f"Halt: {second.halt_reason}")
print()
print(second.output[:600])

The iterations counter continues where the first run left off — no re-work.

2. The goal queue

SchedulerDaemon owns a JSON file at ~/.shipit_agent/autopilot-queue.json. Any process can enqueue(); the daemon drains them on its tick.

In [ ]

python

from shipit_agent.scheduler_daemon import SchedulerDaemon

# llm_factory builds a fresh LLM per run — important for long daemons
# where tokens/credentials may rotate.
daemon = SchedulerDaemon(llm_factory=lambda: llm, tick_seconds=3)

daemon.enqueue(
    run_id="nightly-lint",
    objective="Summarise all 'error' lines in build.log",
    success_criteria=["Counts per file reported", "Top 3 noisiest files listed"],
    budget={"max_iterations": 4, "max_seconds": 180},
)
daemon.enqueue(
    run_id="morning-status",
    objective="Write a two-paragraph status for yesterday's engineering work.",
    success_criteria=["Mentions merges", "Mentions open blockers"],
    budget={"max_iterations": 3},
)

for entry in daemon.list_queue():
    print(f"{entry.run_id:<20} [{entry.status:<7}] {entry.objective[:60]}")

3. Drain one at a time — `run_once()`

For CI or a cron job that should fire one goal per invocation:

In [ ]

python

result = daemon.run_once()
if result is None:
    print("(queue empty)")
else:
    print(f"{result.run_id} → {result.status} (iters={result.iterations})")

# Look at the queue afterwards — status moves from 'pending' → 'done'/'halted'.
for entry in daemon.list_queue():
    print(f"{entry.run_id:<20} [{entry.status}]")

4. Run forever — `run_forever()`

In a notebook we won't block on this, but the one-liner you'd use in production is:

daemon.run_forever()

It installs SIGINT/SIGTERM handlers, ticks on tick_seconds, drains each pending goal, and emits daemon_heartbeat events on idle. Systemd / launchd / Docker can supervise it directly.

Equivalent CLI: shipit daemon --tick 5 (see shipit_agent.cli_autopilot).

5. Clean up the queue

In [ ]

python

for run_id in ("nightly-lint", "morning-status"):
    daemon.remove(run_id)
print("queue length:", len(daemon.list_queue()))

Summary

Autopilot writes a checkpoint after every iteration.
resume(run_id) picks up at the last successful iteration — no re-work.
SchedulerDaemon drives multi-goal 24h operation. Queue state lives on disk; the daemon process is stateless.

Next: 40, 41, 42 — specialist agents in action.

Persistence, resume, and the scheduler daemon

Its imports and arguments are checked against shipit-agent 3.0. The outputs were saved when it was written, for an earlier release.

In [ ]

python

from pathlib import Path
import sys

# Resolve the repo root whether this cell is running from ./notebooks
# or from the repo root — mirrors the existing 01–36 notebook series.
ROOT = (
    Path.cwd().resolve().parent
    if Path.cwd().name == "notebooks"
    else Path.cwd().resolve()
)
if str(ROOT) not in sys.path:
    sys.path.insert(0, str(ROOT))

from examples.run_multi_tool_agent import build_llm_from_env

# Bedrock Llama 4 Scout is the default model this library ships with
# (see `shipit_agent/config.py`). No model name required — the helper
# reads `AWS_REGION` / credentials from env and returns an adapter that
# works with every Autopilot / Agent class.
llm = build_llm_from_env("bedrock")
print("Bedrock LLM ready:", type(llm).__name__)

1. Crash → resume round-trip

We'll simulate a crash by capping max_iterations=1 for the first run, then resume with a larger cap.

In [ ]

python

from shipit_agent.autopilot import Autopilot, BudgetPolicy
from shipit_agent.deep import Goal

goal = Goal(
    objective="Outline a simple CLI todo app architecture.",
    success_criteria=[
        "Files listed",
        "CLI commands described",
        "Storage choice justified",
    ],
)

# First run — tiny budget, expected to halt partial.
first = Autopilot(
    llm=llm,
    goal=goal,
    budget=BudgetPolicy(max_iterations=1, max_seconds=120),
).run(run_id="todo-arch")

print(f"First run → {first.status} (iters={first.iterations})")
print(f"Halt: {first.halt_reason}")

Now resume — Autopilot loads the checkpoint and keeps going.

In [ ]

python

second = Autopilot(
    llm=llm,
    goal=goal,
    budget=BudgetPolicy(max_iterations=6, max_seconds=240),
).resume("todo-arch")

print(f"Resumed → {second.status} (iters={second.iterations})")
print(f"Halt: {second.halt_reason}")
print()
print(second.output[:600])

The iterations counter continues where the first run left off — no re-work.

2. The goal queue

SchedulerDaemon owns a JSON file at ~/.shipit_agent/autopilot-queue.json. Any process can enqueue(); the daemon drains them on its tick.

In [ ]

python

from shipit_agent.scheduler_daemon import SchedulerDaemon

# llm_factory builds a fresh LLM per run — important for long daemons
# where tokens/credentials may rotate.
daemon = SchedulerDaemon(llm_factory=lambda: llm, tick_seconds=3)

daemon.enqueue(
    run_id="nightly-lint",
    objective="Summarise all 'error' lines in build.log",
    success_criteria=["Counts per file reported", "Top 3 noisiest files listed"],
    budget={"max_iterations": 4, "max_seconds": 180},
)
daemon.enqueue(
    run_id="morning-status",
    objective="Write a two-paragraph status for yesterday's engineering work.",
    success_criteria=["Mentions merges", "Mentions open blockers"],
    budget={"max_iterations": 3},
)

for entry in daemon.list_queue():
    print(f"{entry.run_id:<20} [{entry.status:<7}] {entry.objective[:60]}")

3. Drain one at a time — `run_once()`

For CI or a cron job that should fire one goal per invocation:

In [ ]

python

result = daemon.run_once()
if result is None:
    print("(queue empty)")
else:
    print(f"{result.run_id} → {result.status} (iters={result.iterations})")

# Look at the queue afterwards — status moves from 'pending' → 'done'/'halted'.
for entry in daemon.list_queue():
    print(f"{entry.run_id:<20} [{entry.status}]")

4. Run forever — `run_forever()`

In a notebook we won't block on this, but the one-liner you'd use in production is:

daemon.run_forever()

It installs SIGINT/SIGTERM handlers, ticks on tick_seconds, drains each pending goal, and emits daemon_heartbeat events on idle. Systemd / launchd / Docker can supervise it directly.

Equivalent CLI: shipit daemon --tick 5 (see shipit_agent.cli_autopilot).

5. Clean up the queue

In [ ]

python

for run_id in ("nightly-lint", "morning-status"):
    daemon.remove(run_id)
print("queue length:", len(daemon.list_queue()))

Summary

Autopilot writes a checkpoint after every iteration.
resume(run_id) picks up at the last successful iteration — no re-work.
SchedulerDaemon drives multi-goal 24h operation. Queue state lives on disk; the daemon process is stateless.

Next: 40, 41, 42 — specialist agents in action.

Persistence, resume, and the scheduler daemon

1. Crash → resume round-trip

2. The goal queue

3. Drain one at a time — run_once()

4. Run forever — run_forever()

5. Clean up the queue

Summary

Persistence, resume, and the scheduler daemon

1. Crash → resume round-trip

2. The goal queue

3. Drain one at a time — run_once()

4. Run forever — run_forever()

5. Clean up the queue

Summary

3. Drain one at a time — `run_once()`

4. Run forever — `run_forever()`

3. Drain one at a time — `run_once()`

4. Run forever — `run_forever()`