Data preparation for custom models: selecting, sanitising and recording what a model learns from

A model trained on a few thousand carefully chosen, correctly labelled examples usually beats one trained on a large pile of whatever was easy to export. The model cannot tell a reviewed answer from a hurried one, a current policy from a withdrawn one, or a fact someone confirmed from a guess someone typed. It learns all of them with equal confidence.

So most of the work in a custom model is choosing what goes in and being able to say why. That work also produces the evidence a reviewer, an auditor or your own team will ask for later: which records were used, under what basis, with what removed, and how the evaluation set was kept apart. If you cannot answer those questions, the model cannot be defended, however well it scores.

Sanitisation happens before training, not after. The pre-model sanitisation gateway, which is in progress, sits between stored content and any model and removes what must not reach one. The brain already refuses to read or store passwords and other credentials, and removes values that look like secrets before anything is stored. Personal and restricted data are local-only by default.

For training data the gateway is designed to go further: strip or replace direct identifiers, mask account numbers and similar values, and drop records that are flagged restricted for this purpose. Each removal is recorded, so the datasheet can say what was taken out and by which rule. Until the gateway is finished, teams preparing data with their own pipeline should apply the same idea: sanitise first, and keep the log of what was removed.

Whether a given set of records may be used to train a model is a decision for your organisation and its legal team. It depends on what the people concerned were told, what basis the data was collected under, the jurisdiction and the purpose. Swfte does not give legal advice and does not decide this for you.

What the platform is designed to do is record the decision: the purpose a dataset was assembled for, the lawful basis your team recorded, and any consent conditions that apply, so the record travels with the dataset and the model trained on it. Consent capture is on the roadmap. The goal is technical controls and evidence that help an organisation meet its own obligations, with the posture depending on use case, jurisdiction, deployment and configuration.

Every dataset should leave with a datasheet: a short document that says where the records came from, the date range, the selection rule, what sanitisation removed, the evidence statuses accepted as labels, the lawful basis your team recorded, the split method and the decontamination check. It also names who assembled it and who approved it. A hash of the final files ties the datasheet to exactly what was trained on.

The datasheet belongs in the model's lineage record, next to the base model and version the Model Vault records and the evaluation results that justified promotion. When someone asks months later why the model behaves the way it does, the answer starts here.

The brain supports per-person export and erase commands and optional retention that pseudonymises deleted people; legal holds on content are part of its knowledge core, which is in progress. A held record must not be deleted; an erased record must not come back. Both affect training. A dataset that contains a person later erased is no longer a dataset you can rebuild as it was.

For a trained model the honest position is that removing one person's influence from trained weights is not something current methods do reliably. The practical answer is to rebuild the dataset without the erased records and retrain, then evaluate and promote the new version. Plan for this from the start: keep the pipeline repeatable, keep the datasheet, and decide in advance how quickly a retrain follows an erasure request.

The brain does not export training data today; export for training is on the roadmap. What you can do today is prepare data with your own pipeline, following the steps on this page, train with your own tooling, and host the result in the Model Vault, where the base model and versions are recorded and the endpoint sits behind Connect. Supported adaptation methods for a managed service are <supported adaptation methods - founder to fill>.

A worked example. An internal service desk wants a model that drafts first replies in its own format. The team takes resolved tickets whose resolutions a lead had confirmed, drops anything with restricted fields, masks names and account numbers, removes near-duplicates and splits by ticket thread. It sets aside the most recent month as the evaluation set and checks no training example matches it. The datasheet records every step and the basis the legal team approved. When a former employee later asks to be erased, the team rebuilds the dataset without those tickets and retrains.