<retrieval · agents · evals · cost>
Garrett John Law
I build LLM systems that survive contact with production.
Retrieval that stays grounded, agents that stay in bounds — and the eval harnesses, tracing, and cost controls that make them shippable rather than demoable.
Three of them are running right now. These systems run in production; the numbers trace to the source and runtime config behind them.
On the defended side of the AI line
Generic application work is commoditizing fast, while agent and LLM-infrastructure work is where demand is heading. These systems are how I stay on the right side of that shift — I do the work, not just claim it.
At Re:Build Manufacturing I build AI infrastructure (Terraform, observability, RAG). On my own time I run a sales pipeline of ~15 LLM agents (plus a few supporting Lambdas), a 135-tool operations agent, and an automated website-evaluation engine — all in production, all on Bedrock.
~15
Autonomous agents in production
135
Tools in one agent’s registry
$100→$50
Monthly Bedrock cap cut (spend roughly halved)
<case-studies />
AI systems, in production
Not demos. Three systems where LLMs do real work behind guardrails — running unattended, with the numbers below tracing to the source and runtime config behind them.
Production multi-agent system
A self-improving fleet of LLM agents that runs a sales pipeline
An autonomous lead-generation system of ~15 LLM agents (plus a few supporting Lambdas) on AWS Lambda + Bedrock that discovers prospects, drafts grounded outreach, and rewrites its own strategy every week.
~15
LLM agents (plus supporting Lambdas)
$100→$50
Monthly Bedrock cap cut (spend roughly halved)
Weekly
Self-rewritten strategy brief
Problem
A one-person company cannot afford to spend hours each week prospecting, and naive LLM outreach is worse than none: it hallucinates criticism of a prospect’s website and burns money re-drafting the same held leads on every run. The system had to be cheap, self-correcting, and safe to send unattended.
AI architecture
- Structured as three layers — Sensors (discovery, analytics, event-enrichment) collect from Brave Search, Google Maps, and GA4; Synthesizers (the weekly strategy optimizer, daily pulse, weekly report) read the history and write a strategy brief; Dispatchers (outreach, content, demo-dispatcher) act on leads.
- A grounding gate sits in front of every send: a Claude Haiku evidence-auditor classifies each draft against the research it was built from, surgically repairs the one offending clause once, and HOLDS the draft rather than sending an unsupported claim — four blocking checks (unsupported-defect, premise-contradiction, diagnostic-subject, unsolicited-criticism).
- The strategy optimizer reads ~90 days of conversion history and uses Bedrock tool-use to emit a fresh strategy brief every Sunday; discovery and outreach read that brief at startup, so scoring multipliers, email experiments, and volume self-adjust to what actually converted.
- Two model tiers keep cost down — Haiku for scoring and validation, Sonnet only for the writing that matters. Concurrency is held by an S3 compare-and-set run-lock (the account is below the Lambda reserved-concurrency floor). Ships via GitHub Actions OIDC to a leadgen/prod branch.
Measured result
The pipeline runs fully unattended Monday–Friday; the founder only touches warm replies. Diagnosing a self-contradicting drafter prompt (it rewarded confident specificity with no gate to the evidence) and replacing the blind regenerate-cascade with a single surgical repair fixed the dominant cause of held drafts. Cutting the monthly Bedrock cap from $100 to $50 (in config), plus a 72-hour re-draft dedup window, roughly halved observed outreach spend.
Production agent + context engineering
A 135-tool operations agent with one source of truth for its rules
A Bedrock Claude agent that runs a whole operation — bookkeeping across three entities, real-estate underwriting, CRM, calendar and domain workflows — through 135 tools, with a "doctrine" system that single-sources its domain rules across the agent prompt, the Claude Code skills, and the code itself.
135
Agent tools in one registry
~76%
Per-turn tool-token cut via on-demand discovery
1
Source of truth for domain rules
180s
Async worker beats the 30s gateway cap
Problem
The same domain rules (how to route an expense, when a receipt links to a transaction, which calendar an event belongs to) lived in three places at once — the agent’s system prompt, a dozen Claude Code SKILL.md files, and the code — and silently drifted apart. And a conversational agent that calls real tools cannot fit inside an API Gateway’s 30-second response window.
AI architecture
- A doctrine module defines each domain contract exactly once as a structured fragment. renderDoctrine([...]) composes the agent’s system prompt from those fragments at load time; a generator stamps the same fragments into the Claude Code skill docs between markers — so editing one file updates every surface that consumes the rule.
- The agent’s entire tool surface (135 read/write tools) is declared once in a shared tool-registry: read tools auto-execute and feed results back to the model; write tools are surfaced with full schemas but intercepted and staged as reviewable plan-ops that only commit after the human approves.
- Loading all 135 tool schemas every turn would cost ~27K tokens; instead the agent loads only a curated hot set and discovers the long tail on demand via a search_tools meta-tool (ranked over the registry, with a per-conversation activated set) — a ~76% per-turn cut. Anthropic’s native tool-search defer-loading is InvokeModel-only on Bedrock and this agent runs on the Converse API, so it’s a hand-rolled equivalent of their tool-search guidance.
- To beat the 30s gateway cap, /chat/send writes a "thinking" status row and self-invokes the Lambda asynchronously (InvocationType: Event); the real agent loop runs off the request path with a 180s timeout while the UI polls status. Up to 20 model turns per conversation.
- A receipt-to-transaction matcher runs daily: it links a paid receipt to its ledger transaction on an exact-amount, ±7-day, unique match (vendor keyword as tie-breaker), and the link is enqueued by the canonical writer itself — the contract is code-enforced, not documented and hoped for.
Measured result
Domain rules stopped drifting because there is now structurally one copy. The agent runs long, multi-tool reasoning turns without hitting the gateway timeout, and proposes writes that a human approves before they ever touch live records — the assistant operates the system without being trusted to act unsupervised.
Automated evaluation engine
An evaluation engine that grades any website against a fixed standard
The Full Score™ — a quality-grading engine that audits a live site across six weighted pillars, runs real browser-rendered checks, and turns the gaps into a prioritized plan.
6
Weighted quality pillars
7
Automated checks, static + rendered
A–F
Grade on a 0–100 score
Problem
Most website audits grade one dimension (speed, or SEO) and stop. Worse, static accessibility checkers are blind to the real failure modes — they can only compare text against a solid CSS background color, so light text laid over a busy hero image passes a static check and fails a human. Selling a website rebuild needs a defensible, repeatable standard, not a vibe.
AI architecture
- Six weighted pillars (Brand & Design, Message & Persuasion, Technical Foundation, Integrations & Workflow, Maintainability & Ownership, Conversion & Growth) are codified as the single source of truth; seven automated checks (accessibility, SEO, links, content, structured-data, performance, mobile) map onto them and roll up to a 0–100 score and an A–F grade.
- A rendered (real-browser) mode adds what static tools cannot see: it screenshots the page with and without text to measure true text-over-media WCAG contrast, click-tests CTAs to catch buttons that do nothing, renders at a 390px phone width to find overflow and sub-44px tap targets, and measures per-image weight against explicit standards.
- The output that matters is not the letter grade — it is the prioritized list of gaps. The grade is the diagnostic lens; the gap vocabulary is the deliverable.
- Wired into the sales motion: the same engine grades a prospect’s legacy site and the proposed rebuild side by side, so the pitch is a before/after scorecard rather than an opinion.
Measured result
A repeatable, code-enforced website standard that produces a client-facing scorecard on demand — catching the exact text-over-image contrast failures that pass every static checker, and converting an audit into a numbered to-do list a prospect can act on whether or not they ever hire.
What I build with
The capabilities behind the systems above — the work AI can’t do for itself.
Multi-agent orchestration
Fleets of LLM agents organized as sensors, synthesizers, and dispatchers — running unattended on serverless infra.
LLM safety & grounding gates
Evidence-auditor gates and surgical-repair loops so models never ship an unsupported claim. Fail closed, hold for review.
Context engineering
Single-source domain contracts that compose the agent prompt, the tooling, and the skills — so rules never drift.
Human-in-the-loop tool agents
Agents with large tool surfaces that propose writes and commit only after a human approves. Operate, don’t override.
Automated evaluation
Rubric-driven scoring engines that grade real artifacts — including browser-rendered checks static tools can’t see.
Bedrock production infra
Claude on AWS Bedrock, Lambda, DynamoDB, SES, Terraform — model-tiered for cost, deployed through CI/CD.
This site ships a grounded assistant
The assistant in the bottom corner does keyword retrieval over a verified fact corpus — the same grounding discipline I’d feed a Bedrock model (answer from source, or say you don’t know). It’s a demo, not a live model call.
Open it and ask how the Bedrock cap was cut, or what the doctrine system does.
Building with AI? Let’s talk.
I’m open to AI/agent engineering and AI-infrastructure roles, and to consulting through Fugue Lab. If you need someone who ships AI systems that actually run, reach out.
Get in touch