How to evaluate an open-source LLM

An evaluation without a pass rule is a demo. Open a text file and write four things before you run anything: the job the model must do and for whom, the constraints (data class, region, hardware, languages, latency), the baseline you compare with (a hosted API, your current model, a rule-based system), and the result that lets a candidate go forward.

Make the pass rule specific. "At least 85 per cent of my tasks pass, no task in the safety group fails, and the 4-bit build is within three points of the full-precision build" is a rule. "Good enough" is not. Choose the numbers from the cost of a wrong answer in your job, not from what the first model happens to score. Setting the rule after you have seen results is how evaluations bend to fit the model someone already likes.

Keep this note. It becomes the first page of the decision record in step 8. The longer checklist version of this step is in our open-weight evaluation checklist.

Do this early because it is free and decisive. Open the model page, find the license field in the model card metadata (the Hugging Face model card documentation describes the field, and says a custom licence is declared as license: other with a name and link), and open the licence file that ships with the checkpoint. Read the one that applies to this exact version. A family often changes terms between releases.

Two examples show how different they can be. The Apache License 2.0 grants a perpetual, worldwide, royalty-free copyright licence to reproduce, modify, sublicense and distribute the work. Redistributing it means giving recipients a copy of the licence and marking modified files with prominent notices. It also ends your patent licence if you sue over the work. The Gemma Terms of Use (last modified 1 April 2026 when we read them) work differently. They incorporate a Prohibited Use Policy, require a notice file and a copy of the terms when you redistribute a model derivative, say Google claims no rights in outputs, and reserve Google's right to restrict use it reasonably believes violates the agreement. The same page says Gemma 4 is covered by its own licence, which it describes as Apache 2.0, so check which release you actually hold.

Write down the licence name, the URL, the date, and the commit hash or file checksum of the checkpoint. If your plan involves redistribution, fine-tuning for a customer, or serving users in a regulated sector, get a lawyer to read the text. This is orientation, not legal advice. For the supply-chain side of the same question, see model supply chain security.

Public benchmarks tell you how a model does on someone else's questions. Your task set tells you how it does on yours. Collect 30 to 50 real inputs from the job, remove personal data, and for each write what a good answer must contain and what it must not contain. Put at least five tasks in a safety group (requests the model should refuse or deflect), and add any language you serve beyond English.

Keep the rules mechanical where you can: required phrases, forbidden phrases, a JSON field that must exist. Mechanical checks are cheap, repeatable and honest about their limits. Use a human reviewer for what cannot be checked that way, and read at least ten raw answers yourself. If you plan to use another model as a judge, calibrate it against your own labels first; our guide to validating AI systems covers the pitfalls.

Save the tasks as one JSON object per line. The scorer below sends each prompt to an OpenAI-compatible server at temperature 0, checks the rules, and prints the pass rate per group. It uses the openai Python package pointed at a local base URL, which is how the OpenAI SDK is redirected to any compatible server.

llama.cpp ships llama-server, which exposes POST /v1/chat/completions and POST /v1/completions and has a GET /health check. Start it with the model file and a port. Set the context size with -c to what your tasks need, and offload layers to a GPU with -ngl if you have one. The server listens on 127.0.0.1 by default; leave it that way while you test.

To get the GGUF file, convert the original Hugging Face checkpoint with the convert_hf_to_gguf.py script from the llama.cpp repository. The llama.cpp quantisation README shows the command with a 16-bit output type, and notes that --outtype auto or leaving it out also works when the model is distributed in 16-bit. If a ready-made GGUF is published by the model's own authors, you can use that, but then record where it came from.

Keep this as your reference run. Every other number in this guide is a comparison with it.

Most self-hosted deployments run a quantised build, because it fits in less memory and usually runs faster. Quantisation can cost accuracy, and the llama.cpp quantisation README says the loss is usually measured as perplexity or KL divergence and can be reduced with an importance matrix. Do not assume the loss is small for your task: measure it.

Create the quantised file with llama-quantize, giving the input file, the output file and the type. Q4_K_M is the example type in the README, and the README shows results for many others, from IQ1_S up to Q8_0 and F16. Start a second server on a different port and score the same tasks against it. If your hardware cannot run both servers at once, run them one after the other.

Compare per task, not only in total. A build that loses two points overall may have broken exactly the tasks you care about. List the tasks that changed from pass to fail and read them.

EleutherAI's lm-evaluation-harness runs standard benchmarks against many backends. Its README says it covers over 60 standard academic benchmarks. Use it for two jobs: to catch a build that is broken in a way your small task set cannot show, and to compare with published numbers for the same task. Do not use it to choose the winner; your task set does that.

Install from source as the README shows, plus the backend extras you need (lm_eval[api] for API-based models). For a llama.cpp server the README gives a GGUF form that takes the server URL. Use --output_path to save results and --log_samples to keep every input and output for later inspection; the interface documentation says --output_path is required with --log_samples. List the tasks you can run with lm-eval ls tasks.

The harness documentation says --limit is for testing only. Use it to check your command works on a small slice, then run the full task for any number you will quote. Run the same task, the same few-shot count and the same settings for both builds, or the comparison means nothing.

A model that passes your tasks but answers in a minute is not usable. llama-bench is the benchmark tool in llama.cpp. Its README gives the form llama-bench -m model.gguf -p 512 -n 128 -r 5 -ngl -1 -o md, where -p is the prompt length, -n the number of generated tokens, -r the repetitions and -ngl -1 offloads all layers to the GPU. The output table reports pp (prompt processing) and tg (token generation) rows in tokens per second with a standard deviation.

Choose -p and -n close to your real requests. A chat reply of 200 tokens after a 3,000 token prompt behaves differently from the defaults. Your task scorer already prints the median request time, which includes everything the user waits for; keep that number as well.

For memory, start from arithmetic. The weights take roughly the parameter count multiplied by bytes per weight, so 8 billion parameters need about 16 GB at 16 bits and about 4 to 5 GB at 4 bits. On top of that comes the context cache, which grows with the context size you set and the number of requests you serve at once. Read the real figure from your operating system or GPU monitor while the server handles a long prompt, and record the peak, not the idle value.

Open-weight models differ widely in how they handle refusals and instructions hidden in input. Your task set already holds a safety group; widen it with the cases that matter to you: requests for data the model should not have, instructions embedded in a document it is asked to summarise, and prompts that try to change its role. Keep each case as a task with a mechanical rule, and read every failure.

If you serve users in several languages, run the core tasks in each of them and report the pass rate per language. A model that scores well in English and drops sharply in Portuguese or Korean is a different model for those users. Do the same for long inputs: send one task at the longest context you plan to support and check the answer still uses the end of the input.

Do not treat a clean result here as a security review. It shows the model is not obviously unsafe on your cases. For adversarial testing of the whole application, use how to red team an LLM.

Write the result down while you still remember what you ran. A decision record is what you show a colleague, an auditor or your future self when the model is questioned. Include the pass rule from step 1, the result against it, the licence you read, the exact files and checksums, the command lines, the hardware, and the date.

Add a retest trigger. Models, runtimes and your own tasks all change. Retest when you change the model, the quantisation, the runtime version or the prompt, and on a fixed schedule even if nothing changed. Keep the task file and the results in version control next to the code that uses the model.

If you will serve the model to users, the next job is to put it behind a proper endpoint. See how to self-host an LLM. If the result is a fail, that is a useful outcome: you learned it for the price of an afternoon.