8 micro-topics in Reliability & Operations, each with the evidence that proves you have it and the written reason behind every link. Before you start, it rests on 6 other domains. Downstream, it holds up 4 domains.
8 topics · depth 0–9 of 16 · 8 internal links · 6 in · 5 out
Load-bearing links first. Each one says what in this domain rests on what outside it, and why.
1 link
Gate changes on evals rests on Automate workflows with CI
The gate runs in CI.
1 link
Recognize silent LLM failure rests on Reason about nondeterministic output
Variation is why fluent output can be wrong.
1 link
Use a model as a judge rests on Get structured output
A judge returns a structured score.
1 link
Sequence evals by cost rests on Treat the context window as a budget
Both are resource allocation decisions.
1 link
Build an evaluation dataset rests on Reason about the testing pyramid
An eval set is a test suite for a nondeterministic system.
1 link
Write deterministic checks rests on Generate JSON Schema from models
Structural checks are the same validation applied to output.
In prerequisite order. Each idea names the artifact that closes it, the market demand that put it on the map, and what it stands on.
Depth 0 · meta · scaffolded · M22 §22.4
Samples output for review where it matters
Human labels are the reference the judge is measured against.
Depth 4 · conceptual · unassisted · M22 §22.4
Knows fluent output can be wrong
Reason about nondeterministic output · Model Foundations
Variation is why fluent output can be wrong.
Depth 5 · representational · unassisted · M22 §22.4
Collects representative cases with expected outcomes
You build a dataset because you cannot trust a single result.
Reason about the testing pyramid · Testing Foundations
An eval set is a test suite for a nondeterministic system.
Depth 6 · procedural · unassisted · M22 §22.4
Asserts on structure, format and constraints
Checks run over the dataset.
Generate JSON Schema from models · Validation & Pydantic
Structural checks are the same validation applied to output.
Depth 8 · conceptual · scaffolded · M22 §22.4
Scores output with a rubric-driven model
You reach for a judge only when cheap checks cannot decide.
Get structured output · Model Integration
A judge returns a structured score.
Depth 9 · meta · scaffolded · M22 §22.4
Checks the judge against human labels
A judge must be checked before it is trusted.
Depth 9 · meta · scaffolded · M22 §22.4
Blocks a regression before it ships
Gating requires a score.
Automate workflows with CI · Delivery & CI/CD
The gate runs in CI.
Depth 9 · meta · scaffolded · M22 §22.4
Runs cheap checks before expensive ones
The ladder orders checks by cost.
The expensive rung is the judge.
Treat the context window as a budget · Context Engineering
Both are resource allocation decisions.
Everything downstream that names an idea here as a prerequisite, grouped by where it lives.
2 links
Measure retrieval quality rests on Build an evaluation dataset
Metrics are computed over an evaluation set.
Measure generation faithfulness rests on Use a model as a judge
Faithfulness is scored by a judge.
1 link
Add an independent checker rests on Write deterministic checks
A checker needs a definition of correct.
1 link
Measure and mitigate bias rests on Build an evaluation dataset
Fairness metrics run over an evaluation set.
1 link
Reflect on results rests on Write deterministic checks
Self-evaluation borrows from how you score output.
Retool. Retrain. Relaunch.
375 ideas. 17 weeks. No tuition, ever.
Vets Who Code is a veteran-run 501(c)(3). The accelerator is free, remote, and we don’t take a share of your first paycheck.
Free · Remote · 17 weeks · EIN 86-2122804