Evaluation and safety for custom models: nothing is promoted until it has passed

A model version moves through stages: development, staging, production. Evaluation is what decides whether it may move. That sounds obvious, but the common failure is the opposite order: a team ships a model because it looked good in a handful of demonstrations, then builds the evaluation after the first complaint. By then there is no baseline to compare against and no record of what was known at the time of release.

The rule here is simple. Before a candidate is promoted, it is run against the same suites as the model in production, on the same cases, and the results are kept. A candidate that does better on the domain task and no worse on safety and general behaviour can move on. One that does not stays where it is, whatever the deadline.

A regression gate compares the candidate with the model currently in production on every suite. If the candidate is worse beyond the margin you set in advance, on any suite you have marked as blocking, promotion stops. The gate blocks; it is not a warning that someone can click past. Changing the margin is itself a reviewed decision, recorded with the reason.

Red-teaming adds people trying to break the model on purpose, with reference attack tooling alongside them. It finds failures that fixed suites miss, and every finding becomes a new case in the suite. Be exact about what a clean run means: no known attack succeeded during that run. It does not mean the model is safe in any absolute sense, and a report that says so should not be trusted.

A base model's refusal behaviour is itself the product of training, and further training can disturb it. Published research has found that fine-tuning on entirely benign data can weaken refusals, and that a small number of harmful examples can weaken them a great deal. Adapters are not exempt. Quantising the weights for serving can shift behaviour too, most visibly outside English.

So every adaptation is treated as a new model. A new adapter, a new training run on refreshed data, a new quantisation of the same weights: each goes back through the full safety suite and the regression gate, and the artifact that is tested is the artifact that is served. Swfte calls this safety-first adaptation, and it is the design intent behind Swfte Safety, published at /models/swfte-safety as a design-intent model card. That card publishes no measured results, and we say so plainly.

The approach on this page is the one Swfte publishes for testing open-weight models at /open-source-model-testing: your own tasks, a safety suite and an EU-language suite, run against the candidate, with the result deciding whether it goes further. A custom model is held to the same method as any open-weight model you would deploy, plus the comparison with the model it replaces.

The named domain suites Swfte would provide are <named evaluation suites - founder to fill>. Until those exist, the suites are yours, built from your work, and the method is what keeps them honest. That is no hardship: a suite built from your own cases tells you more about your model than any public benchmark, and it stays with you whichever model you run next.

Today you can score single-turn chat in Studio's evaluation, put guardrails and content policy in front of a model at the Connect gateway, with secret and personal-data detectors and redaction, and follow the published testing method. An agent evaluation sandbox with recorded, replayable traces is in progress as internal tooling. Named domain suites, an automated red-team pipeline and gated promotion wired to evaluation results are on the roadmap.

A worked example. A claims team has a candidate adapter that drafts summaries. It passes the domain suite clearly. On the safety suite it holds refusals in English but declines fewer harmful requests in two other EU languages than the production model does. The gate blocks promotion. The team adds multilingual safety examples to training, retrains, and runs everything again. The second candidate passes, a named reviewer signs the evidence pack, and the model moves to staging.