Prompt Engineering Is Not a Skill, It's a System: How We Actually Build Prompts
How we build production prompts: versioned templates, evaluation sets, structured outputs and model changes handled as releases, not wordsmithing.

"Prompt engineer" was briefly one of the most talked-about job titles in tech. The idea was that the right words, found by the right person, would make a model behave. That works for demos. In production, prompts that depend on one person's feel for wording break quietly, and nobody notices until a customer does.
What works is treating prompts like any other part of the software: versioned, tested, reviewed and released.
The problem with prompts as craft
A prompt tuned by hand usually has these properties:
- Nobody knows why it works. The wording that fixed one case might have broken three others.
- It is not versioned properly. It lives in a string in the code, or in a dashboard.
- Changes are judged by feel, often on a handful of examples the author already had in mind.
- Model upgrades break it, and nobody finds out until users complain.
None of that is a wording problem. It is a process problem.
The evaluation set comes first
Everything else depends on this. Collect 50 to 200 real inputs from the task, and for each, write the output a competent person would accept. Include the easy cases, the hard ones and the ones that have gone wrong before.
Then every change can be measured. Did this edit improve the score? Which categories got worse? Is the new model better, or just different? Without an evaluation set, those questions are answered by opinion.
How we score outputs
Different outputs need different checks:
- Structured fields: exact match or schema validation.
- Rules: required elements present, forbidden content absent, length within bounds.
- Reference similarity: how close an answer is to an accepted one.
- Model-graded rubrics for open-ended writing, with a clear rubric and regular human spot checks to keep the grader honest.
We report scores by category, not just overall. An average that rises while one category collapses is a warning sign.
Prompts as versioned templates
Prompts live in version control as templates with variables, next to the code that uses them. Each version is reviewed like code and linked to the evaluation results that approved it. That gives you history, blame and rollback, the same as any other change.
Structured outputs
When a prompt must return data, define a schema: required fields, types and allowed values. Validate every response. If it fails, retry with the error, or fail visibly. Parsing free text with regular expressions is one of the most common causes of silent production bugs.
Break big prompts into steps
A single prompt asked to classify, extract, reason and write in one go is hard to test and harder to debug. Splitting it into steps lets you:
- evaluate each step separately;
- use a smaller, cheaper model for simple steps;
- see exactly where a failure happened;
- change one step without retesting everything.
Better inputs beat better words
Most quality problems we fix are input problems: missing context, irrelevant retrieved passages, or examples that do not match real cases. Improving what goes into the prompt, through retrieval, context selection and well-chosen examples, usually moves quality more than rewording the instructions.
For how retrieval fits in, see fine-tuning vs prompting vs RAG.
Model changes are releases
Model providers update and retire models regularly. We treat every model change as a release:
- Run the evaluation set on the new model.
- Compare scores by category.
- Adjust prompts where needed and re-run.
- Release to a small share of traffic and monitor.
- Keep the ability to switch back.
Silent upgrades in production are how quality drops without anyone noticing.
Monitoring in production
We log the input, output, prompt version and model version for each request, with personal data redacted. A sample of outputs is reviewed by people each week. Schema failures, user feedback and escalations are tracked. Every real failure becomes a new evaluation case, so the same mistake is caught before the next release.
What about tools?
Version control, a simple evaluation script and good logging cover most teams' needs at first. Specialist prompt management and evaluation tools help at scale, especially when non-engineers edit prompts. None of them replaces the evaluation set; they just make it easier to run.
A minimal prompt system you can set up this week
You do not need special tools to start:
- A folder of prompt templates in your repository, one file per task, with variables for inputs.
- An evaluation file per task: inputs and accepted outputs, starting with 50 real examples.
- A script that runs every example through the current prompt and model and scores the results.
- A rule that no prompt change merges without an evaluation run attached to the pull request.
- Logging of prompt version and model version on every production request.
That is enough to turn prompt changes from guesswork into measured releases. Add tooling later when the number of prompts or editors grows.
Signs your prompts need a system
- Nobody can say which version of a prompt is in production.
- A model upgrade caused problems that took days to notice.
- Fixes for one type of input keep breaking another.
- Prompt changes are tested on a few examples in a chat window.
- Outputs are parsed with fragile string matching.
Start with the evaluation set
Our first fixed-scope project for prompt work is two weeks: we build an evaluation set from your real inputs, score your current prompts, move them into versioned templates, and deliver the first measured improvement. For agent workflows built on top of these prompts, see agentic workflows in 2026.
We build production AI systems for teams across India, the United States, the United Kingdom, the UAE and Canada; see AI and ML services.
Prompt systems, in numbers
Blurbee's assessment of relative impact on production quality, scored out of 10. Opinion based on our engagements.
Why teams hand prompt engineering to Blurbee
We treat prompts as code: versioned, reviewed, tested against an evaluation set and released like any other change. The result is quality you can measure and changes you can roll back.
- Fixed-scope first projectTwo to four weeks, priced as a number rather than an estimate, so you have evidence before a longer commitment.
- One delivery lead, namedYou always know who to ask. No account manager relaying questions to a team you never meet.
- Code and IP are yoursAssigned in writing, handed over with repositories, infrastructure definitions and documentation whether or not you continue with us.
- Performance budgets in CICore Web Vitals and bundle size gate the build. Slow does not ship.
- Accessibility to WCAG 2.2 AAReviewed with automated and manual passes, not claimed on a slide.
- Written overlap disciplineStanding calls inside your working window, everything else asynchronous with a decision log.