MLOps
Models are software. Run them like it.
The pilot worked; production is a different sport. We build the operations layer — deployment, monitoring, evaluation, retraining — that keeps models and LLM applications honest after launch.
What we build
- Deployment pipelines for models and LLM apps
- Production monitoring: quality, drift, latency, and cost
- Evaluation as routine, not ceremony
- Update and retraining loops with human review where it matters
Proof
Our own hiring
The agent that writes our candidate profiles has run in production for a year without changes — instrumented, evaluated on real output, and left alone because the log stayed quiet.
Read the field note- A year in production
- No changes needed
When teams come to us
- A model that works in a notebook nobody dares deploy
- Quality degrading silently until a customer notices
- AI costs that surprise finance every month
How we build it
Baseline and metrics
define what "good" means before touching anything
Deployment pipeline
models and LLM apps ship like software
Monitor what matters
quality, drift, latency, cost
Update loop
retraining and prompt changes with human review
What degrades silently
Nothing in production breaks loudly. It drifts:
Quality
the inputs change and the model does not. A product release changes what tickets say; a season changes what the camera sees. Accuracy decays while the dashboard stays green, because nobody is comparing outputs to a sample of truth every week.
Prompts and versions
a prompt edited to fix one case breaks three others; a provider updates a model under the same name. Without an evaluation set run on every change, "it seems fine" is the only signal, and it is not one.
Latency
a feature that answered in two seconds now takes six, one retrieval step at a time. Users stop using it before anyone files a ticket.
Cost
per-call pricing scales with success. The feature that works is the one that surprises finance, and the fix is a budget per tenant that the product respects, not a spreadsheet in arrears.
The pilot proves the model can work. Operations is proving, every week, that it still does.
What running it well looks like
Every model or LLM application we operate has the same four things attached, and they are attached before launch, not after the first incident: an evaluation set of real cases with known-good answers, run automatically on every change to a prompt, a model or the data; a deployment pipeline where a model version ships like a code release — reviewed, tagged, reversible; monitoring on the four things that degrade, with thresholds decided with the team that will get the alert; and an update loop where retraining or a prompt change goes through a person before it reaches production.
The unglamorous truth is that most of this is ordinary software discipline applied to a component that happens to be probabilistic. Teams that already ship software well are most of the way there; what is usually missing is the evaluation set — the one artifact nobody builds in the pilot and everybody needs in production.
Works well with
FAQ
We only use LLM APIs — do we still need this?
Yes: prompts, versions, evaluations, and costs degrade silently too.
Can you operate models our team built?
Common case: we instrument first, then improve.
What does running AI cost?
Depends on volume — the practice is making cost visible and reviewed like code, so the answer is never a surprise.
What is an evaluation set, and why does everything come back to it?
A few dozen to a few hundred real inputs with the answer a competent person would give, stored next to the code and run on every change. It is the only way to know whether a change improved the system or moved the failures somewhere you were not looking. Without it, every prompt edit is a guess.
How much does monitoring cost to run?
A fraction of the model's own cost — the sampling that catches drift compares a slice of outputs to truth, not every call. The expensive thing is not monitoring: it is the customer who notices first.
Do you work with open-weight models on our infrastructure?
Yes, and the operations layer is the same shape either way: versions, evaluation, monitoring, a reversible pipeline. What changes is that cost becomes a capacity decision instead of a per-call one, and the model can be updated on your schedule rather than a provider's.