A prototype demoing well in a slide is a different animal from a system carrying real traffic. We design for the second, and we tell you honestly which one you have.
Most of the LLM systems we have been asked to review in the last two years were prototypes that had graduated to production without anyone noticing. They looked fine when the demo user asked the demo question. They produced quietly incorrect answers, or answers that referenced tools that had been removed, or answers that leaked customer data into the wrong tenant, once the traffic profile turned into what real customers actually ask.
The gap between the two is not usually about the model. It is about the architecture around the model — the retrieval strategy, the tool interface, the eval harness, the observability, the safety net that catches the answer before it reaches a paying user. That work is unglamorous. It is also the work.
What we produce
- System design document. A written architecture — retrieval, prompt structure, tool surface, memory, orchestration, fallback paths — that a competent engineering team can build against without further consulting hours. Not a diagram; a document with sentences and trade-offs.
- Eval framework. A concrete evaluation setup — golden dataset shape, offline metrics, online metrics, red-team prompts, regression guardrails — that produces a number the CEO can read and the engineers can act on. Includes a written definition of what “good enough to ship” means for the specific system, agreed before the eval runs.
- Guardrail & safety design. The specific input filters, output filters, tool-permission model, and human-in-the-loop escalation paths for the system's actual failure modes — not a generic policy from a template. Where the safe answer is “refuse and escalate,” we design the refusal and the escalation path.
- Reference implementation review. If your team is already building, we sit with them for a working session or two, read the code, run the evals, and produce a written review of what is production-shaped and what is still prototype-shaped. Blunt and specific.
What we don't do
We don't ship the code. We are not writing the production pipeline for you, and we are not the ones on the on-call rotation. We design the system, we set the evals, we review the code, and we tell you when it's ready for real users. Anyone who tells you they can be both the architect and the on-call engineer is trying to sell you an engagement that never ends.
We also don't chase model releases. If your team wants to switch to whatever model launched last Tuesday because it topped a leaderboard, we will ask what changed in your evals to justify the switch. Usually the answer is nothing changed in your evals, because there weren't any. That is the actual work.
An eval framework nobody looks at is worse than no framework at all — it produces a false sense of ground truth. We design evals a specific person on your team owns, at a specific cadence, with a specific decision the eval is intended to inform. If nobody would act on the number, we don't spend the time to produce it.
Engagement
Systems design is typically project-shaped — a three-to-eight-week design and evaluation phase, occasionally followed by a light retainer for quarterly design reviews as the system evolves. Fee structures are fixed-fee for the design phase; retainers cover a fixed number of hours and a direct line for design decisions between reviews.
Introductions happen by referral. If you were pointed here by someone we work with, mention their name when you write. If not, tell us plainly what you're building and what has to be true for it to be considered working. We reply within two business days.