AI Model Evaluation in Toronto: How We Actually Test a Model Before It Ships

How Toronto fintech and health-tech teams should test AI models before launch: metrics, calibration, fairness, robustness, OSFI E-23, PHIPA and costs.

AI Model Evaluation in Toronto — Blurbee Solutions

Toronto's Financial District and the health-tech companies around MaRS share a common rule: nothing important goes live on a demo. A bank's model risk function, a hospital's privacy office or an insurer's compliance team will ask how a model was tested, and "it looked good on our examples" does not pass.

This is how we evaluate a model before it ships, whether it is a classic predictive model or a large language model feature.

Start from the decision

Evaluation begins with the decision the model supports, not the model. Who acts on its output? What happens when it is wrong in each direction? A missed fraud case and a blocked legitimate payment have very different costs, and so do a missed clinical flag and a false alarm. Those costs choose the metrics and the thresholds.

The six checks

1. Performance on realistic data

Test on data the model has never seen, drawn from conditions it will actually face: later time periods, new branches or clinics, current customer mix. Random splits of historical data often overstate performance.

2. Calibration

If the model says 80% likely, is it right about 80% of the time? Poorly calibrated scores lead to bad thresholds and to people over- or under-trusting the model.

3. Subgroup performance

Overall metrics hide where a model fails. We break results down by groups and conditions that matter: regions, product lines, age bands, language, device types, and protected characteristics where they can be lawfully and reliably used. Large gaps get investigated before launch, not after a complaint.

4. Robustness

How does the model behave with typos, missing fields, unusual values, noisy images or inputs slightly outside its training range? Fragile models fail quietly in production.

5. Safety and policy, for generative models

For LLM features we add adversarial tests: prompt injection, attempts to extract data the user should not see, unsafe or off-brand content, and confident answers without supporting sources. These are scored like any other metric.

6. Drift readiness

Before launch we agree what will be monitored: input distributions, output distributions and, where outcomes arrive later, real accuracy. Without that, a model degrades silently. See MLOps from notebook to production.

Evaluating language models

LLM evaluation uses an evaluation set of real inputs with accepted outputs, scored by schema checks, rules, similarity to reference answers and model-graded rubrics with regular human spot checks. Every prompt or model change reruns the set. For building these sets, see prompt engineering as a system.

Regulatory expectations in Toronto

Banking and insurance. OSFI's Guideline E-23 on model risk management has been revised to cover AI and machine learning models. Federally regulated institutions should expect governance, independent validation, monitoring and documentation proportionate to each model's risk. An evaluation report built for that audience saves weeks in review.

Health. In Ontario, PHIPA governs personal health information held by custodians and their agents. Evaluation should use de-identified data where possible, stay inside approved environments, and involve the privacy office early.

Privacy generally. PIPEDA applies to commercial personal information, including data used for testing.

What the report contains

  • The decision, intended use and out-of-scope uses.
  • Data used for evaluation, and how it differs from training data.
  • Results by metric, subgroup and condition, with uncertainty.
  • Failure analysis with examples.
  • Recommended thresholds and their trade-offs.
  • Known limitations.
  • Monitoring plan.
  • The evaluation code and data references, so every future model change can be retested.

Rates in Toronto

Toronto agencies typically charge CA$90-175 per hour for mid to senior engineering. Our blended AI team is CA$41-82. A pre-launch evaluation of one model typically takes two to four weeks. We keep 3.5 hours of live overlap with Eastern time.

A worked example: a credit decision model

A Toronto lender wants to launch a model that recommends approve, review or decline for small-business loans.

The evaluation tests it on the most recent two quarters, not a random sample. Calibration shows the model is overconfident in the middle of its range, so thresholds are adjusted. Subgroup analysis by region and business age shows weaker performance for businesses under two years old, so those go to manual review by default. Robustness tests reveal the model reacts badly to missing financial statements, so the input form is changed to require them.

The report goes to model risk with the evidence for each decision. The model launches narrower than first planned, and safer.

Start with one model

The first fixed-scope project evaluates one model against its real use and delivers a report your risk or privacy team can review, plus a reusable test suite. For the related Toronto stack, see when Angular still wins.

Model evaluation in Toronto, in numbers

E-23OSFI's guideline on model risk management, revised to cover AI and machine learning modelsOffice of the Superintendent of Financial Institutions
PHIPAOntario law governing personal health information held by custodians and their agentsPersonal Health Information Protection Act, 2004 (Ontario)
6 checksPerformance, calibration, subgroups, robustness, safety and drift readiness, before any model shipsBlurbee evaluation framework
CA$90-175Typical Toronto agency hourly band for mid to senior engineeringComposite of Toronto agency rate cards, 2026
What a pre-launch evaluation typically finds, in our reviews

Blurbee's assessment of the most common issues found in pre-launch model evaluations, share of findings. Indicative.

  • Weak performance on a subgroup ~30%
  • Test data not like real use ~25%
  • Poor calibration of scores ~18%
  • Fragile to small input changes ~15%
  • Unsafe or off-policy outputs ~12%
Share of findings

Why Toronto teams evaluate models with Blurbee

We test models against the decision they support, the people they affect and the conditions they will meet, and produce evidence a bank's model risk function or a hospital's privacy office can actually review.

  • Fixed-scope first projectTwo to four weeks, priced as a number rather than an estimate, so you have evidence before a longer commitment.
  • One delivery lead, namedYou always know who to ask. No account manager relaying questions to a team you never meet.
  • Code and IP are yoursAssigned in writing, handed over with repositories, infrastructure definitions and documentation whether or not you continue with us.
  • Performance budgets in CICore Web Vitals and bundle size gate the build. Slow does not ship.
  • Accessibility to WCAG 2.2 AAReviewed with automated and manual passes, not claimed on a slide.
  • Written overlap disciplineStanding calls inside your working window, everything else asynchronous with a decision log.

Frequently asked questions

Frequently asked questions

What does AI model evaluation involve?

Why isn't a single accuracy number enough?

How do you evaluate a large language model?
What is calibration?
How do you test for fairness?
What does OSFI expect for AI models?
Does PHIPA affect model evaluation?
How large should an evaluation set be?
How much does model evaluation cost in Toronto?
What do we receive?

Start with a fixed-scope first project

Two to four weeks, priced as a number. Code, accounts and IP stay in your name.