How to create your own local model

Pick an instruction-tuned open-weight model in the 0.5B to 4B range for a first run. Smaller models train faster, fit on modest hardware and make mistakes you can see quickly. Read the licence on the exact checkpoint you will use, because your tuned model inherits it.

The table shows what the Hugging Face model API reported for these checkpoints on 6 October 2026. Gated means you must accept terms before you can download. If a licence is anything other than Apache-2.0 or MIT, read the licence text itself before you build on it, and ask your legal team if the model will be sold or shipped to customers.

This guide uses Qwen/Qwen3-4B in its examples, which is listed under Apache-2.0. Swap in any model whose architecture your chosen tool supports; check the tool documentation, not the model card, for that.

Write each example as one line of JSON in chat format: a list of messages with the final one from the assistant. This is the conversational format both TRL and mlx-lm read, so the same files work on every path in this guide.

Quality beats quantity. Include the hard cases and the inputs that currently fail, not only easy ones. If two reviewers would label the same input differently, settle the rule in writing before training, or the model learns the disagreement. Strip personal data and anything you have no right to train on.

Split before you do anything else, and never look at the test file while tuning. The script below shuffles with a fixed seed and writes an 80, 10, 10 split. It is our script, not from any tool documentation. If the same customer, document or conversation appears in several examples, split by that unit instead, otherwise near-duplicates leak from train into test and flatter the result.

Without a baseline you cannot say the tuning helped. Install Ollama, pull the same base model, and score it on the test file. On macOS and Linux the install is one line; on Windows use the PowerShell line. Ollama serves an OpenAI-compatible API on port 11434, so a short script can send each test prompt and compare the reply with the reference.

The script uses exact match, which suits classification and fixed formats. For free text, replace the comparison with a rubric or a human review; the guide how to validate your AI covers that, including the traps of using a model as the judge. If your model prints reasoning text before the answer, strip it before comparing. Keep temperature at zero so reruns are comparable.

Record the number and the date. If the base model already clears your bar, stop here: a better prompt is cheaper than a tuned model you now have to own.

QLoRA loads the base model in 4-bit and trains small adapter matrices on top, which is why it fits in far less memory than training in 16-bit. The Hugging Face docs describe it as quantising the model to 4 bits and then training it with LoRA. TRL, PEFT and bitsandbytes are the standard open-source stack. The script below follows the documented TRL pattern: pass a peft_config and a quantization_config to SFTTrainer, use target_modules="all-linear" as the PEFT quantisation guide advises for QLoRA, and use a learning rate near 1e-4, which the TRL docs suggest for adapters.

Treat the epoch count, batch size and rank as starting points, not truths. Watch the validation loss. If it falls and then climbs while training loss keeps falling, you are overfitting: stop earlier, use fewer epochs or add data. Unsloth's guide recommends one to three epochs.

If you prefer a faster, lower-memory wrapper, Unsloth installs with uv pip install unsloth --torch-backend=auto and publishes notebooks for specific models. Axolotl is configuration-driven: you write one YAML file and run axolotl train. Both need an NVIDIA GPU for this path; Axolotl asks for Ampere or newer for bf16 and Flash Attention.

On a Mac, mlx-lm is the simplest route. It trains LoRA by default, and if you point it at a quantised model it trains QLoRA. Your three JSONL files go in one directory named by --data, with the names train.jsonl, valid.jsonl and test.jsonl.

The mlx-lm notes list the model families its LoRA training supports (Mistral, Llama, Phi2, Mixtral, Qwen2, Gemma, OLMo, MiniCPM, InternLM2 at the time of reading). If your base model is newer than that list, run mlx_lm.lora --help and try a short run before committing a day to it; if the architecture is not supported the command will tell you early.

To cut memory, lower --batch-size (default 4), lower --num-layers (default 16), add --grad-checkpoint, or shorten examples. The same notes report about 250 tokens per second on an M1 Max with 32 GB for a 7B model with batch size 1 and 4 layers on a sample dataset. Use that only as a rough guide: one million training tokens at 250 tokens per second is about 67 minutes. Run a few steps and read your own rate.

A LoRA adapter is a small file that sits on top of the base. To convert to GGUF you first need one ordinary model directory, so merge the adapter into the base weights. With PEFT this is merge_and_unload(). The PEFT docs are explicit that it is not an in-place operation: you must keep the model it returns.

Load the base in 16-bit for the merge, not in 4-bit, so the merged weights are not rounded twice. This needs enough memory for the full-precision model, which is about 8 GB for 4B parameters, but it can run on CPU if the GPU is too small.

Other tools have their own merge step. Axolotl uses axolotl merge-lora with the same YAML and the adapter directory. Unsloth has save_pretrained_merged with save_method="merged_16bit". On a Mac, mlx_lm.fuse --model <your-base-model> loads adapters from adapters/ and writes fused_model/; its GGUF export is limited to Mistral, Mixtral and Llama-style models in fp16, so for other families run the fused model with mlx-lm or train on an NVIDIA machine for the GGUF route.

GGUF is the file format llama.cpp and Ollama read. Conversion is two phases: convert the merged Hugging Face model to a high-precision GGUF, then quantise that file. Quantising from 16-bit or 32-bit gives better quality than re-quantising something already quantised.

Get llama.cpp from its repository and build it with CMake. Install its Python requirements, run convert_hf_to_gguf.py on the merged directory, then run llama-quantize with the method name. Q4_K_M is the usual first choice. The llama.cpp notes show Llama 3.1 8B at 14.96 GiB in F16 and 4.58 GiB at Q4_K_M, and 7.95 GiB at Q8_0. Their measured perplexity table is how you would pick a different method if Q4_K_M loses too much quality on your task.

The llama.cpp notes warn that some newer models need a newer transformers than its requirements file installs, and say pip install -U transformers is safe. If conversion fails with an unknown architecture, update llama.cpp before anything else.

Unsloth users can skip the manual route: model.save_pretrained_gguf("directory", tokenizer, quantization_method="q4_k_m") does the merge and quantise in one call. Unsloth also warns that the most common cause of gibberish is a different chat template at inference than at training.

Ollama does not quantise GGUF files on import, which is why you did it in the previous step. Create a Modelfile that points at the file, build the model with ollama create, and then run the same evaluate.py against it. Compare the number with the baseline from step 3. If the tuned model beats the base by the margin you set in advance, and nothing else you care about got worse, it passes.

If you would rather use llama.cpp directly, llama-server -m my-model-Q4_K_M.gguf starts an OpenAI-compatible server; the server notes say it listens on 127.0.0.1:8080 by default. Point the evaluation script at http://localhost:8080/v1/ to score it the same way.

Also test what you did not train on. Ask general questions, ask things you want it to refuse, and try inputs in other languages if you serve them. Our post on safety-first fine-tuning lists what to check. For a fuller method covering judges, regression gates and an evidence pack, see how to validate your AI.

A tuned model is a new artefact someone has to own. Write down: the base model and exact revision, its licence and the date you read it, the dataset version and how examples were approved, training settings, the quantisation method, the baseline and tuned scores, and the safety checks you ran. Store the adapter, the merged weights and the GGUF together with a hash of each.

Decide when to retrain: when the task definition changes, when the base model is superseded, or when monitoring shows the score slipping on fresh cases. Decide when to retire it, and who is allowed to say so.

If you need to serve it to more than one person, put it behind a proper server rather than a laptop. How to self-host an LLM covers that. Swfte does not offer managed fine-tuning today; its Model Vault is designed to hold weights you bring, with versions and promotion stages. Details are on the custom models pages linked below.