Reliable recurring work
Evaluation sets, regressions, and maintenance
Keep a workflow reliable as its inputs, instructions, and surrounding products change.
Lesson 3 of 6 in the recommended order · About 25 min (estimate)
On this page
Practical AI glossary — terms and common confusions
- Model
- A learned component used to produce a result.
A drafting model is one part of a letter app; it is not the whole app.
- Application
- The software experience around components and services.
A letter app adds accounts, storage, and a Send button.
- Prompt
- Instructions and input supplied for a task.
“Summarize this notice in two bullets” specifies a task and shape, not a truth guarantee.
- Token
- A unit a model processes; it need not be a whole word.
A tokenizer can split a name into pieces; count using the actual system.
- Context
- Information available for the current request.
An earlier attachment may be absent even when its filename is visible.
- Training
- Adjusting a model using examples.
A training log differs from a conversation correction.
- Inference
- Using a trained model to produce a result.
Drafting another sentence does not by itself prove retraining.
- Retrieval
- Selecting material to supply to a task.
A retrieved notice still needs a relevance and date check.
- Embedding
- A numerical representation used to compare or organize items.
A similar search match is a candidate, not proof of the same meaning.
- Tool
- A function an application can call.
A calendar lookup can be read-only; finding an event is different from creating one.
- Agent
- Software that can choose and carry out steps toward a goal.
An agent still needs bounded permissions and checked action receipts.
- Hallucination
- Plausible generated content that is unsupported or false.
An invented opening time needs evidence or removal, even when phrased confidently.
- Evaluation
- Checking results against stated criteria.
Twenty checked examples establish a bounded result, not universal reliability.
- Multimodal
- Working with more than one kind of input or output.
A photo plus text does not prove every small label in the photo was read correctly.
Outcome
Use a change record and evaluation results to decide what must be retested and what is actually established.
Why it matters
Keep a workflow reliable as its inputs, instructions, and surrounding products change.
Concept
Keep a small versioned evaluation set: representative tasks, difficult boundary cases, expected checks, and the source packet used. Record the prompt, model or application version where relevant, settings, outcome, and review date. This makes a later difference investigateable.
A regression is a previously satisfied requirement that fails after a change. A successful old run does not establish that a new version meets the requirements. Retest changed behavior and important shared paths, investigate failures, and keep the previous usable version available when appropriate. Maintenance also includes refreshing source dates and checking that permissions and owners still apply.
Worked example
A fictional template update improves tone but drops a required exception in an old evaluation case. That is a regression under the unchanged rubric. Restore the exception and retest both the failing case and other cases affected by the template change. Do not remove the inconvenient case to make the report green.
Before you check
Create a maintenance record with version, change, affected checks, results, unresolved issues, and next review trigger. Include at least one correct output that should be accepted unchanged.
Practice and fresh transfer
The packets below are fictional and contain the facts needed for these cases. External references are optional background. Judge each response independently: it may be supported, contradicted, or unresolved. Select the passages needed to justify your judgment and write why the distinction matters before revealing feedback.
Assess the response
Use Accept when all material claims are supported. Use Revise when a supplied fact or requirement is contradicted. Use Evidence is insufficient when a key fact cannot be established either way. If a response contains both an unknown and a direct contradiction, choose Revise and explain both problems. Conflicting claims with no established authority remain insufficient; a claim does not become a governing fact merely because a source asserts it.
Some responses are fully supported. Others need correction or more evidence. Judge each on its sources; do not edit a correct answer just to change it.
Your written notes stay in this page and disappear when you leave. Only a self-reviewed completion can be saved to your learning account. These practice checks do not establish independent proficiency.
Practice
Case 1
Consider a fictional template regression report.
Source packet
- Source 1
- Version record: V1 preserved the written-extension exception. V2 omitted it on the same case.
- Source 2
- Rubric: The exception remains required.
Response to assess
V2 regressed on the exception requirement and needs repair before acceptance under this rubric.
Practice
Case 2
Consider a fictional status report for an updated system.
Source packet
- Source 1
- Test log: V1 passed 12 cases last month.
- Source 2
- Complete current status record: V2 was installed today and has not yet been run on these 12 cases.
Response to assess
V2 has passed all 12 cases.
Fresh transfer
Case 3
Consider a fictional evaluation-set maintenance choice.
Source packet
- Source 1
- Set record: Eight ordinary cases and two missing-source cases are relevant to the workflow.
- Source 2
- Update result: The new prompt fails one missing-source case.
Response to assess
Keep the relevant failing case, investigate the failure, and rerun affected checks after repair.
Fresh transfer
Case 4
Consider a fictional prediction from unchanged past results.
Source packet
- Source 1
- Record: The workflow passed its tests six months ago.
- Source 2
- Current state: Source policies, input formats, and application version have not been checked since then.
Response to assess
The workflow will still satisfy every current requirement.
Save your self-review
Completion records that you reviewed the cases. Your explanation and transfer performance need a facilitator to establish independent learning.
Sign in with your learning-center account to save completion.
Review the explanation for every case before saving.
Summary and next step
Apply the checklist to a new task. Preserve supported content, explain any change with evidence, and name what remains unresolved. Saving records self-review, not independently demonstrated proficiency. A facilitator must assess the explanation and fresh transfer for human learning evidence.