How to audit AI systems

Pick one system, not “our AI”. Name its boundary: the model or models, the prompts, the retrieval data, the tools it can call, the people and systems it touches, and the environments (development, staging, production). A boundary that leaves out the tools or the data is the usual way an AI audit misses the real risk.

Write the audit question in one line: “Does the claims-triage assistant operate under effective controls for data access, accuracy, change and human oversight between 1 January and 30 September?” Then set the period you will sample from and the date evidence closes.

Record independence. If you helped build the system, say so and have someone else lead the testing. Agree who receives the report and what happens to critical findings. Get the access permissions in writing now; waiting for them in week three is the most common delay.

You cannot test what nobody has listed. Ask for the AI system inventory and check it against reality: cloud bills for model API spend, gateway logs, identity-provider app grants and a short survey of teams. NIST AI RMF GOVERN 1.6 describes an AI system inventory as an organised database of artefacts relating to an AI system or model, which may include system documentation, incident response plans and data dictionaries. Use that as a minimum for what each entry holds.

For the system in scope, draw the data flow: where inputs come from, which model processes them, where outputs go, what is stored and for how long, and which third parties see the data. Note every third-party model, dataset and package; NIST MAP 4 and GOVERN 6 both ask for risks from third-party software and data to be covered.

Ask the owner to confirm the map in writing. Differences between the owner’s map and what you find in logs and configurations are your first findings. The AI estate inventory post gives a sample schema, and how to detect shadow AI covers finding systems nobody told you about.

An audit compares what exists with a criterion. Start with the organisation’s own AI policy, because that is what management has committed to. Then add external criteria that apply. Do not try to test everything: choose the ones that match the system’s risk and the reason for the audit.

Keep the matrix to one row per control objective, with the source clause beside it, so every finding can quote the criterion it breaks.

Ask for evidence in one list so nothing is requested twice. Number each item, say what period it covers and who should provide it. Ask for records the system produces by itself (logs, change tickets, approvals) in preference to documents written for the audit.

The list below is a starting point. EU AI Act Annex IV is a useful checklist even outside the law: it names the system description, the development process, monitoring and control, appropriateness of performance metrics, the risk management system, lifecycle changes, standards applied and the post-market monitoring plan.

Start with accountability. NIST GOVERN 2 expects accountability structures in which the right teams and individuals are empowered, responsible and trained. Check that the named owner exists, knows they are the owner, and has authority to stop the system. Check that approval for go-live was given by someone with that authority before the date the system went live, not after.

Then test change control by sampling. Take the list of every change in the period to the model version, prompts, retrieval data and tool permissions. Pick a sample across change types and across months, not only the easy ones. For each, check the ticket, the approval, the test result before release and whether the production version now matches what was approved.

The test that finds the most is comparing the live configuration with the approved one. Ask an engineer to show the running model identifier, prompt version and tool list on screen, in front of you. Differences between that and the last approved change are findings.

A system you cannot reconstruct cannot be audited. For each sampled interaction, ask the owner to show the trail from input to outcome: who or what made the request, which model and prompt version ran, what data was retrieved, which tools were called with which arguments, what was returned, and whether a human approved or changed anything.

EU AI Act Article 12 says high-risk systems must technically allow automatic recording of events over the lifetime of the system, covering events relevant to risk situations, post-market monitoring and operation. Article 19 requires providers to keep the logs they control for a period appropriate to the intended purpose, at least six months unless other law says otherwise, and Article 26(6) sets the same minimum for deployers. Check the retention setting against that, and against your own policy and data protection duties; keeping personal data in logs longer than needed is its own finding.

Use a script to draw a reproducible sample from an exported log, then trace each selected record end to end. The script below fixes the random seed so the sample can be repeated. Record any interaction where a link in the chain is missing.

Do not accept test results on trust. Ask for the eval set and the configuration, then re-run a subset yourself and compare. If the system owner’s reported pass rate cannot be reproduced within a reasonable range, that is a finding about the testing process, whatever the score. The method for building and judging those tests is in how to validate your AI.

For security, scope the testing from the OWASP Top 10 for LLM Applications: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. Review the red-team report if one exists, check that fixes were re-tested, and run a few probes of your own. See how to red team an LLM.

EU AI Act Article 15 asks high-risk systems to be resilient against attempts by third parties to alter their use, outputs or performance by exploiting vulnerabilities, and it names data poisoning, adversarial examples and model flaws. Use that as a prompt for what to ask about.

Oversight on paper is easy. Test it by looking for the traces. Article 14(4) of the EU AI Act lists what a person assigned to oversight must be enabled to do: understand the system’s capacities and limitations, remain aware of automation bias, interpret its output, decide not to use it or to override or reverse its output, and intervene or stop it. Article 26(2) says deployers assign oversight to people with the necessary competence, training and authority.

Ask for training records for the overseers. Then look at approval and override records: if reviewers approve every item in seconds, or never override anything, oversight may be a rubber stamp. Ask someone to demonstrate the stop procedure. For agents, check that actions listed as needing approval cannot run without it. See how to set up human approval for AI agents.

Write each finding in five parts: criterion (what should be true, with the clause), condition (what you found, with the evidence reference), cause (why, if you know), effect (what could go wrong) and recommendation. Use the rating scale agreed at the start. Quote evidence by reference number, not by memory.

Share the draft with the system owner before the report is final and record their response next to each finding. Factual errors get corrected; disagreement about risk is recorded, not erased. Be specific about what you did not test and why. An audit that implies full coverage is worse than one that states its limits.

Agree an owner and a date for each action, then re-test before closing. Closing a finding on a promise is how the same issue appears in the next report.

Track each action to closure and re-test the control, not only the document. Feed confirmed gaps into the inventory and the risk register. Set triggers for an earlier review: a new model, a new tool with write access, a new data source or a serious incident.

Keep the working papers with the report: criteria matrix, evidence list, sample records, scripts and seeds, interview notes and the management responses. A reviewer should be able to follow your conclusions back to evidence a year later.