Deploy and serve a custom model on infrastructure you control

The serving engine decides how many requests one GPU can carry. For shared serving, engines such as vLLM and SGLang keep the card busy with continuous batching; Ollama and llama.cpp suit development and single-team tools. The trade-offs are set out on the deploy-models hub, and the choice is made per deployment against the model, the hardware and the traffic.

Quantisation shrinks the weights so a model fits on less memory, and it changes the model. The evaluation you ran on full-precision weights says nothing certain about the 8-bit or 4-bit file you actually serve, especially on refusal behaviour and on languages other than English. Treat each quantised artifact as a new candidate: re-run the evaluation on exactly the file you will deploy, and record its hash. If you serve an adapter on top of a base, test that combination. How Swfte serves adapters is <how adapters are served - founder to fill>.

Connect is one OpenAI-compatible API in front of many providers. A custom endpoint sits behind it like any other model, so agents and applications call one address and the gateway decides where each request goes. That makes routing policy, not code: send the high-volume, low-risk work to your custom model, keep a fallback for when it is unavailable, and change the split by configuration.

Routing is also the rollback lever. Keep the previous approved revision deployed and routable until the new one has run on live traffic for long enough to trust. If something goes wrong, rolling back is a routing change and a log entry, not an emergency redeploy. Once you are confident, undeploy the old revision and archive it in the vault.

Today usage, cost and latency are visible through the gateway's usage and analytics reads, and the audit stream records what was served. That is enough to see when a model is slow, expensive or suddenly busier than planned, and usage caps can act on it before the budget is gone. It is not the same as knowing whether the answers are still good.

What is not built yet is automatic detection of quality drift: noticing that a model's answers have become less accurate as the work, the vocabulary or the underlying facts change, and raising a retraining trigger. That is designed for and on the roadmap. Until then, run your domain suite on a schedule against the production model, sample live outputs for review, and treat a falling score as a reason to plan the next training run.

For European organisations the default should be that weights, prompts, outputs and logs stay in the EU, on infrastructure where you know who operates it and under which law. Dedicated, in-region hosting is the stated direction of the platform; the regions and facilities on offer are <EU regions and facilities offered - founder to fill>. The guide to deploying an LLM in the EU explains the difference between residency and sovereignty, and why a region label alone is not enough.

Hosting in region is a technical control, and a record of where each model version ran is evidence. Together they help an organisation meet its own obligations; they do not settle them. The posture depends on use case, jurisdiction, deployment and configuration, so decide the hosting option with the people who own those obligations, and keep the decision with the model's lineage record.

A legal operations team has trained an adapter on its own infrastructure and merged it into a small open-weight base. It uploads the merged weights to the Model Vault, which records the hashes and the base model, and creates version one in development. It quantises to 8-bit for serving, uploads that as a separate version and runs the evaluation suite on the quantised file. The results hold, so the version is promoted to staging, then to production after sign-off, and deployed to a dedicated endpoint.

In Connect, the team routes contract-summary requests to the new endpoint and keeps the general model it used before as the fallback. Usage caps are set for the workspace and the model. Some weeks later a reviewer notices weaker answers on a new contract type; the team routes that request class back to the fallback while it prepares a retrain. Hosting, versioning, promotion, deployment, gateway routing and audit are built. Managed training, drift detection and adapter serving are not.