ModelOps
about the project
ModelOps helps teams shipping AI agents decide whether a release is ready. Teams had evaluation results, but no shared record of why a release was approved, what risks remained, or who accepted them.
I reframed ModelOps from an evaluation dashboard into a release-decision system, connecting evaluation evidence, risk ownership, and approval. As the sole product designer, I owned end‑to‑end design across the dashboard, design system, and product landing page, and wrote UI code for the engineering team.
Built at DigitalVertex for a client under NDA. The public demo is an anonymized implementation I designed and built.
Role
Sole product designer
Team
5 engineers, 1 PM
client
DigitalVertex client · NDA
year
2025
Duration
3 months
Status
Closed beta
Release blocked: refund safety below threshold; mandatory review incomplete
the case in 60 seconds
Problem
After every eval run, someone decides whether an AI agent ships. That call was made from a screenshot in a thread and recorded nowhere.
Read→
The shift
The product didn’t need another evaluation dashboard. It needed a release-decision system.
Read→
My role
Sole product designer, end to end: research, IA, UX/UI, design system and handoff, with five engineers and a PM.
Read→
Key decisions
No aggregate score: verdicts per category. A validity check before any number appears. An override that needs a written reason and a name.
Read→
Outcome
In closed beta.
Read→
brief
Initial ask
One tool for teams shipping AI agents: run an evaluation, compare the candidate against production on quality, cost, latency and safety, find the regressions, and decide before release. Eight named screens, desktop-first, one end-to-end scenario.
Business objective
Teams needed to tell whether a new version was actually better, and to sign off before it reached production. The brief set one North Star: time to a release decision the approver could defend.
↳ Kept. Measured as decision time plus whether the decision was right.
Constraints & scope
01In scope: comparison, verdicts, ownership, sign-off, and one screen that reports back whether the gate was right.
02Out of scope: drift monitoring, incident review, a mobile dashboard (the product UI is desktop-first; only the landing is responsive), billing, onboarding, account settings and admin.
Success criteria
01Decision time: from a finished evaluation to a recorded Release, Block or Investigate.
02Override rate: how often a blocked release is approved anyway.
03Production outcomes: whether the release the gate allowed behaved as the run predicted.
All three are logged from the first beta release.
the problem
A run finishes and the summary metric moves by a point or two. The engineer cannot tell noise from a real change without opening the cases; the lead who signs has fifteen minutes and no context from two weeks of work. Neither of them can see which category moved.
Overall safety
Safety on refunds · 640 of 8,000 cases
The overall number passes. The category where the agent moves money fails.
Run #148, candidate v2.4 against production v2.3, same 8,000-case set.
diagnosis
Four reasons the number lied
01
Unaccounted changes break the link to the release
Between two runs the knowledge base, the retrieval index, the judge and the provider’s model can all move. Without a record of those changes, reviewers cannot isolate the effect of the release under review.
02
No shared record of the decision
The reasoning, the named owner and the risks accepted were never written down in one place. Four roles share the call, and two of them open the tool only when a release is waiting, so no one could check later what was agreed, or why.
03
Critical categories need separate review
An improved overall score is not enough to approve a release. Reviewers need to assess critical categories separately so that gains elsewhere do not obscure a local regression.
04
Hypothesis: approving may cost less than refusing
Requiring evidence to block a release, but none to approve it, could bias borderline decisions toward approval. The workflow requires a written reason for both approval and rejection.
in the product
Where the aggregate hid the regression
The aggregate said +0.5%, but 43 cases regressed. This view names the affected cases and their categories, so a reviewer can see what the overall number hides before signing the release.
Regressions view · run #148 against #147, same 8,000-case set.
1
Where it broke, not how much
43 got worse, concentrated in Refunds and not spread across the dataset. The location is the finding.
2
One next step
The only primary action is “Continue human review”. Refunds is a money category, so review is mandatory before the release can move.
3
The judge is checked too
7 cases where the LLM judge and the reviewer disagree get their own filter, so a wrong verdict cannot hide inside a count.
the market
From evaluation results to a recorded release decision.
Five evaluation platforms, checked against their public documentation on 10 August 2026. One question: does the tool carry a run from evaluation results through risk review, a named owner and a recorded approval?
Absent from the docs is not absent from the product
What this meant for the design
No feature is missing from the market. What is missing is a product built around the decision.
Four of the five describe no release gate. The fifth runs it as a binary check inside CI. ModelOps puts evidence, risk review, a named owner and a recorded approval in one workflow. Full monitoring stays out of scope. This is a position, not a moat: any of these tools could close the distance.
how the work unfolded
Designing the release decision, end to end
Six steps, each closed with an output the next one depended on.
01
Examine the evidence
Checked a public roadmap thread from real users, the docs of five platforms, research on LLM-judge reliability and a UX audit of one competitor.
Output
Competitor matrix · graded sources
02
Map decision needs
Grouped the signals into patterns and modeled four roles around one release decision. Every finding links to its source.
Output
Role model · four patterns
03
Define the decision model
Moved the product from showing results to recording a decision. Paired time-to-decision with decision quality, so speed alone could not win.
Output
Reframe · paired success metric
04
Structure the workflow
Mapped the decision flow, roles and IA. Navigation follows the phases of the decision, not data types.
Output
Release-gate flows · role map · IA
05
Standardize interactions
One shell, shared components and a state matrix: blocked, waiting, invalid, not enough data. No state became its own screen.
Output
Design system · 26 screens · 12 dialogs
06
Verify implementation
Automated checks and a screen-by-screen audit found eight discrepancies, three broken MVP commitments. The usability study was set up but not run.
Output
Audit fixes · handoff spec · live build
26 screens designed · 7 shown in 06 Product.
before and after · the two decision models
From a screenshot in a thread to a decision with a trail
The previous workflow spread evidence and approval across screenshots and chat. The redesigned workflow connects comparison results, blockers, ownership, and the recorded release decision.
Before / the decision had no object
01
Screenshot
Environment changes stay invisible
02
Aggregate score
+0.5% overall hid −4.2% on refunds
03
Chat thread
Owner, blockers and conditions not recorded together
04
Approval in the thread
Rationale and criteria not written down
05
No decision record
No durable record for later review
After / the decision is an object
05
Production feedback
What happened, against what the gate predicted
Product manager
→ ProductionProduction findings
inform threshold changes before the next release
Decision flowLoop the product added
users
Four roles, one decision
The engineer is the primary user. The decision is shared with three other roles, and two of them open the tool only when a release is waiting, which shaped where the product has to work.
Engineer · primary user
Engineering lead · approver
Risk owner · refunds
Product manager · consumer
Changes prompts, models and retrieval several times a week. No shared threshold defines safe enough.
Holds the final signature; often decides from a forwarded message.
Owns refund policy. The original scope had left her out.
Ties each version to product metrics; opens the tool only when a release is waiting.
Needs
Needs
Needs
Needs
To know whether a change is safe, and why the numbers moved.
A defensible reason to say no, and a result after saying yes.
Named ownership of the categories where the agent moves money.
An outcome per release instead of a thread.
In the product
In the product
In the product
In the product
Verdicts per category instead of one number, and a validity check before any number appears. → 06 Product
A recorded decision with the reason, the approver and the evidence in one place. → 06 Product
Blockers in money categories are routed to her by name; she accepts or rejects them. → 06 Product
After release, what happened is shown against what the gate predicted. → 06 Product
role interaction
Who initiates, who checks, who signs, who hears back
The author can start a release but never sign it. Production closes the two loops that were missing: the lead hears back, and the engineer learns the result without waiting for a support ticket.
The object · release decision v2.4
State, blockers, owners, reason and trail
Recorded → production outcome
calibrated or underestimated, per metric
01 Initiate
02 Check
03 Sign
04 Hear back
Engineer
Author
Runs the evaluation
—
Cannot sign their own change
Learns the result without a support ticket
Risk owner
Money categories
—
Reviews money cases, accepts or escalates risk
—
—
Engineering lead
Approver
—
—
Final signature. Override only with a written reason
Hears back after release
Product manager
Consumer
—
—
—
Gets feedback per metric
● acts × blocked by design ┆ loop the product added
what the research overturned
Four assumptions I got wrong
Public feedback from real users (a Langfuse roadmap thread, 31 and 8 votes), documentation of five platforms, research on LLM-judge reliability and a UX audit of one competitor. Four roles were modeled from that evidence.
I assumed
What the evidence showed
What the product does now
Statistical significance would be the strongest differentiator
The overall number is the one figure the modeled primary user does not act on, so significance on it changes nothing.
Significance is computed inside each category, never on the overall metric.
Home is where the decision happens
Two of the other three roles open it only when a release is waiting: the lead decides from a forwarded summary, the product manager from a direct message.
The decision became a portable object. Home serves the engineer, who actually lives there.
The baseline is the current production version
Comparisons broke for reasons unrelated to the agent version.
The baseline is a snapshot of seven environment layers.
The risk owner sits outside the scope
Refunds is a money category with mandatory review, and the engineer who wrote the change should not be the one accepting that risk.
Two screens were added: Risk approvals and Policies.
synthesis · how sixteen signals became four patterns
Sixteen signals, four patterns, one system
Four inputs, in this order: a kickoff with the PM and stakeholders on business requirements, technical limits and success criteria; desk research and a competitor review; 12 in-depth interviews with ML engineers and risk owners; product analytics to check what people said against what they did. Every design response traces back through a risk and a pattern to a signal. Tags mark the source.
Tags: interview = in-depth interview (ml eng = ML engineer, risk = risk owner / reviewer) · data = product analytics · pm = PM and stakeholder sessions · roadmap = public user feedback · paper = published research · matrix = competitor docs · scenario = demo data
16 signals
4 patterns
4 risks
4 design responses
Two weeks of green runs, zero cases on the new planSCENARIO
Same set, three runs: 82.1 / 81.6 / 82.4DATA
Judge score up, repeat contacts up about 15%DATA
LLM judges give most answers 3–4 of 5 and are confidently wrong on specialist tasksPAPER
Pattern 01
False calm is worse than a false alarm
Risk
Regressions ship behind green numbers
Design response
Category verdicts, no aggregate, “not enough data” at full weight
Knowledge base edited by another teamINTERVIEW · ML ENG
Retrieval index rebuilds overnightINTERVIEW · ML ENG
Run comparison is a top complaint in a public Langfuse roadmap threadROADMAP · 8 VOTES
Both serious incidents came from untracked layersPM
Pattern 02
Nobody sees what moved between two runs
Risk
Deltas blamed on the wrong change
Design response
A seven-layer validity check before any delta
Saying no to a release has to be justified, saying yes does notINTERVIEW · RISK
A gate that blocks too often gets switched offINTERVIEW · ML ENG
No release gate in 4 of 5 tools; the fifth is a binary pass/fail in CIMATRIX
Nobody wants to own the refunds call aloneINTERVIEW · RISK
Pattern 03
A gate with no legal way around it gets bypassed
Risk
The gate gets switched off
Design response
A governed override and a non-author signature
Engineers never hear how a release turned outINTERVIEW · ML ENG
The engineer learns from support, a week laterINTERVIEW · ML ENG
The PM cannot tie a version to her metricsPM
Two of three roles open it only when a release is waitingDATA
Pattern 04
Nobody gets feedback on the decision
Risk
Nobody gets better at the call
Design response
Production scores the gate and updates thresholds
One system
The release decision as an object: valid, owned, recorded and scored
FlowRiskSystem output
the reframe
The product didn’t need another evaluation dashboard. It needed a release-decision system.
01 · Validate
Seven layers checked before any delta.
In
The candidate and its baseline
Out
Deltas only when nothing but the intended change moved
02 · Detect
Categories, never the average.
In
An evaluation run on 8,000 cases
Out
A verdict per category: improvement, regression or not enough data
03 · Decide
A gate designed from the bypass.
In
Blockers with thresholds and owners
Out
Approve by a non-author, block, or override with a reason
04 · Record
Production reports back to the gate.
In
The decision and its reasons
Out
An audit trail, then a production verdict on the gate itself
↺ Feedback loop
When production contradicts the gate, the miss returns to Policies as a threshold proposal. The gate improves by being wrong.
PRIORITIZATION
44 features, 21 in the build
Every feature was scored against the primary user’s job and sorted MoSCoW. The cut was made on one question: does the release decision survive without it? Three were deliberately refused: not postponed for capacity, but ruled outside the product.
The cut
Does the release decision survive without it?
0
Must · in the build
0
Should · next release
0
Could · later
0
Refused
What was refused, and why
Draft mode: the agent prepares, a human sends
A strong idea raised by the secondary role, but it is a mode of the agent, not a function of an evaluation platform. Building it here would have moved the product’s boundary, not extended it.
Self-hosting
Free on Arize Phoenix and Langfuse, Enterprise elsewhere. It is an infrastructure choice with no interface to design. It would add a sales line, not a product decision.
Billing, onboarding, account settings, admin
Excluded in the brief. These screens add to a screen count but none of them changes whether a release is safe.
21
Must have · in the build
13
Should have
7
Could have
3
Refused
what was refused, and why
Draft mode — the agent prepares, a human sends
A strong idea raised by the secondary role, but it is a mode of the agent, not a function of an evaluation platform. Building it here would have moved the product’s boundary, not extended it.
Self-hosting
Free on Phoenix and Langfuse, Enterprise elsewhere. It is an infrastructure choice with no interface to design — it would add a sales line and no product decision.
Billing, onboarding, account settings, admin
Excluded in the brief. These are the screens a case study adds to raise the screen count; none of them changes whether a release is safe.
the system · the whole pipeline, drawn
The whole system, left to right
Seven stages, five places the release can stop, and one loop back. Each stop is a branch that used to happen silently. Now it is visible, owned and written down.
01 · Run
Evaluation run
v2.4 against production, 8,000 cases
02 · Validate
Validity check
Seven environment layers
03 · Detect
Category risk
A verdict and an interval per category
04 · Decide
Ownership
Every blocker waits on a named owner
05 · Decide
Review
Human review on money categories
06 · Decide
Decision
Approve, block or override
07 · Record
Production feedback
Prediction against outcome
Something else moved: deltas withheld
Interval wider than delta: not enough data
Money category: risk owner required
Unfinished review blocks the release
Author cannot sign: override needs a reason
Misses update the thresholds
the gate learns from being wrong
PipelineLoop the product addedStop conditionDecision object
site map · the same structure, as a drawing
Navigation mirrors the decision
Three groups map to the phases of the decision; the fourth holds the instrument itself. Anything that leaves a trace for someone else opens a centered dialog; everything else stays in a popover.
Show by role:← hover or click to filter
Product
ModelOps / release gate
Evaluate
→ Validate / Detect
HomeENG
EvaluationsENG
Compare runsENG
RegressionsENG
Release
→ Decide
Release review↘ EntryENG LEAD
Risk approvals↘ EntryRISK
Human reviewENG LEAD
PoliciesRISK PM
Eval data
→ The instrument and its outcome
DatasetsENG
Critical casesENG
ProductionPM ENG
Always available
→ Record
Decision historyLEAD PM
Settings
↘ Entry · roles that rarely open the tool land here from a direct link
Overlays follow one rule
Does the action leave a trace someone else will read?
No / anchored popover
Notifications, help, account
Yes / centered dialog
Override reason, new dataset, invite, transfer ownership
design principles
Four rules the whole interface follows
State is stated
Blocked, waiting, invalid and “not enough data” are written in words, never carried by color alone.
Refusals carry full weight
“Not enough data” is set in the same type as a regression. A dash stands where a number would be a lie.
Permissions are state
A disabled Approve says why before anyone reaches for it, not as an error after the click.
Density earns its place
52 px rows and mono numerals for someone comparing 8,000 cases. Slower than a notebook, and it goes unused.
what the literature calls it
Four known failure modes, named
Term
What it means here
How the product answers
Aggregation masking
A composite rose while the category that carries the risk fell.
→ The product declines to compute the composite at all.
Minimum detectable effect
A −1.0% threshold under ±1.1% uncertainty asks the dataset for an effect it cannot resolve.
→ The screen states the contradiction instead of a verdict.
Goodhart’s law
When the judge score became the target, answers grew longer and repeat contacts rose about 15% (scenario data).
→ Four axes, no single target.
Calibrated trust
The risk was not distrust but misplaced confidence.
→ Feedback from production keeps confidence tied to accuracy.
user flow
One release, end to end
The scenario every screen is built on: a blocked release on Home, through review and an override with a reason, to what production did four days later.
01 · Run
Home
v2.4 blocked, 2 blockers
Compare
02 · Validate
Compare runs
Refunds −4.2%, comparison valid
43 cases
03 · Detect
Regressions
43 cases got worse
Open case
04 · Detect
Case detail
The judge said failed
To review
05 · Decide
Human review
12 of 43, counts as a blocker
↓ Back to release review
×Author · blocked
06 · Decide
Release review
2 blockers, risk owner waiting
Can I approve?
07A · Decide
Override reason
A written reason and a name
08 · Record
Decision detail
Recorded with reason, owner and trail
+4 days
09 · Record
Production
Four days later: calibrated or not
Approve is disabled for the author, with the reason written beside it
Not the author
07B · Decide
Approve
Final signature, never the author
Scenario pathAlternativeDialogRecorded object
Inside the product
The screens behind every decision
The scenario every screen is built on: a blocked release on Home, through review and an override with a reason, to what production did four days later.
Compare →
Step 02 of 09 · Validate
There is no aggregate score
The decision
No composite score anywhere, not even as a negation. One axis passing never compensates for another failing.
Why it matters
The first question every user asks is “is it good?”. A single number answers it confidently and wrongly.
What you see
Four axes compared category by category, with a block that names why the overall number misleads: refunds fell 4.2% while the average rose.
decision anatomy
A release decision became an object, not a conversation
Eight fields every release decision has to carry before it can move. If one is missing, the release cannot ship.
Release decision · v2.4 → production
BlockedBefore the call
01
State
02
Blockers
03
Evidence
04
Validity
The call
05
Owners
06
Signature
— missing
07
Reason
— missing · required to override
08
Trail and outcome
2 of 8 fields missing · cannot ship
feedback loop
The product doesn’t end at Approve
A real miss from the scenario: run #147 predicted +90 ms, production did +140 ms. The miss became a proposal in Policies, so the next release is judged by a gate that learned.
01
Release decision
v2.3 approved by the lead, Aug 7
02
Production outcome
Four days live: latency +140 ms
03
Miss detected
Run #147 predicted +90 ms
04
Policy updated
05
Next release
The next candidate is gated by the updated policy
Threshold +300 ms
Predicted (run #147) · +90 ms
Actual (production) · +140 ms
+50 ms miss
0+80+160+240+320
Latency change, ms
The gate learns from being wrong · the next decision uses the updated policy
landing page
The marketing page had to sell a product built on refusals
01
Opens with a question, not a promise.
“Your Evals Ran. Now What?” The headline states the moment the product is for, instead of claiming a score, a verdict or a speed-up the product deliberately withholds.
02
Three numbers instead of customer logos.
7 environment layers compared before a delta is shown, 3 verdicts per category, and 0 numbers on the page that came from anywhere but the product. The page says outright that there are no customers to quote.
03
Every CTA leads into the real product.
Each button opens a specific screen: the release gate, the comparison, the regressions, the decision history. The landing and the dashboard ship as one build.
PRODUCT LANDING PAGE · DESIGNED AND BUILT BY ME
view product landing page
→
Mobile
The approver reads it on a phone
Dashboard: desktop-first · Story: mobile-first
The lead and the PM open the tool only when a release is waiting, usually from a message on their phone. So the page that explains the gate had to work where the decision is read. The dashboard stays desktop-first by design; the story around it does not.
Blocked release, above the fold
The screens behind every claim
From first trace to first decision
“Send this page to your lead”
iteration
Three times the design was wrong first
What triggered it
What it became
01
Significance moved off the headline and into the category
Significance was planned as the main differentiator. But it qualified the overall number — the one figure nobody acts on.
Significance, sample size and interval are computed per category, never on the aggregate. That also made “not enough data” a verdict of its own.
02
The whole interface was rebuilt from light to dark
The first system was light: 19 components, a full token set, 77 instances in the wireframes. The shipped dashboard uses none of them.
The product moved to a dark build: 129 variables, 41 text styles. The light system is archived, not deleted, so the wireframes stay readable as history.
03
A number in a mock was treated as a claim
An impossible figure survived several design reviews because it was in the right format and looked plausible. Nobody checked it, including me.
Every number now traces to the scenario data. The handoff spec separates what research locked from what stays open to visual interpretation.
trade-offs
Three decisions and what each one cost
Deltas are withheld when anything but the intended change moved
What it cost
The most aggressive friction in the product. An engineer who changed one prompt can be blocked by another team’s knowledge-base edit, on a day his change is fine.
The alternative, and why not
Show the deltas with a caveat. Rejected: a number on screen gets acted on, whatever sits next to it. Ships as a hypothesis.
No composite score anywhere
What it cost
Every user’s first question is “is it good?”, and the product refuses to answer in one number. That refusal is paid for in onboarding and in demos.
The alternative, and why not
Show the composite crossed out. Rejected for the same reason: it would still be read. Four axes, no single target, which is also the defense against Goodhart.
The override sits beside Approve, with a written reason and a name
What it cost
A governed bypass is still a bypass. Some releases will go out over a blocker.
The alternative, and why not
No bypass at all. The research had a precedent: a gate with no way around gets switched off, and then there is no record.
outcome
What shipped, and what is still a hypothesis
Shipped
0
Screens on one shell
0
Dialogs and panels
0
Review-queue variants
One shell, one design system, and a handoff spec that separates what research locked from what stays open to visual interpretation. The first page in the file is written for the engineer opening it.
Still a hypothesis
Measuring
—
Time to decision
—
Overrides with a written reason
—
Production calibration
Three decisions ship marked as hypotheses: deltas withheld when anything else moved, no composite score, and the override beside Approve. No outcome numbers are claimed until they are measured.
reflection
What I would carry into the next system
The strongest decision was a refusal
I spent weeks designing better ways to display a difference. The answer was to withhold it until the comparison is proven valid. Next time I start with what a screen must refuse to show.
Speed without a loop breeds confidence
Time-to-decision cannot be optimized alone. It needs a paired measure of whether the decision was any good, or the product optimizes for a worse outcome, faster.
Modeled users are a hypothesis, not evidence
The research was desk-based and the roles were modeled. The next system starts with five real approvers before the first wireframe, not after the last one.
in one line
ModelOps stopped being a place to read evaluation results and became the place where a release decision is made, owned and recorded.
destinations
Explore the demo (live dashboard)View product landing pageNext case study: Interlock
Built at DigitalVertex for a client under NDA. The public demo is an anonymized implementation I designed and built.
Interlock
CONNECT
[ COPY ]
BASED IN
San Francisco · remote · open to relocation
LOOKING FOR
Senior Product Designer · AI Product Designer · UX/UI