How to build an air-gapped AI environment

Write the boundary down before you download anything. Name the networks that count as inside, the single route by which files may enter, and whether anything may ever leave. Most teams allow files in and nothing out except reviewed reports. If you allow nothing out at all, say so, because it changes how you collect logs and error reports.

List the four kinds of thing that will cross: model weights and tokenizer files, container images, Python packages, and operating system updates. Each needs an owner, a checksum record and an approval step. Treat them as a supply chain. A weight file is code you will run on a machine that holds sensitive data, and the file formats differ in risk. the Hugging Face security documentation warns that loading a pickle file can run arbitrary code, and that pickle is the default format for PyTorch weights. Prefer safetensors or GGUF, and load pickled checkpoints only from sources you trust.

Decide how you will tell the people inside that something has changed. In an isolated network nobody reads release notes by accident, so an import ticket should say what was added, why, and which test it passed. That ticket is also the beginning of your evidence pack.

Install the Hugging Face command line on the connected machine. The current documentation recommends a standalone installer, and pip install -U "huggingface_hub" also works. The command is hf, with hf download for fetching.

Use --local-dir to put the files in a folder you control instead of the shared cache. The documentation notes that a .cache/huggingface/ folder is created inside it to track what was downloaded. For a GGUF model you usually want one quantisation, not the whole repository, so name the file or use --include to filter. Pin a revision with --revision so you can say exactly what you imported. Run with --dry-run first to see what would be fetched.

Replace the placeholders below with the repository and file you chose. The repository name is yours to pick. Check its licence page before downloading, and record the commit hash that hf download resolves to.

Pull the images you will run, then save them to tar archives. docker save writes one or several images to a file, and you can pipe it through gzip to shrink it. Pin images by an exact tag or digest, not latest. The vLLM documentation shows vllm/vllm-openai:latest in its example, which is fine for a first try on a connected machine and wrong for a record you will be asked about later.

For Python packages, pip download fetches wheels without installing them. Give it a requirements file and a destination folder with -d. On the isolated side you will install with the package index switched off. The pip documentation says --no-index makes pip ignore the index and look only at --find-links locations, which is exactly the behaviour you want.

Build the wheelhouse on a machine that matches the target: same operating system family, CPU architecture and Python version. pip does have --platform, --python-version and --only-binary options for downloading for a different target, but a matching staging machine avoids a long list of surprises.

Create a SHA-256 manifest of every file in the staging folder with sha256sum, and write the output to a file. On the other side you will check it with sha256sum -c. A manifest only proves the files did not change in transit. It does not prove they were safe to begin with, so keep the manifest, the repository name, the resolved revision and the licence text together in the import ticket.

Store a second copy of the manifest by a different route from the media itself, for example on the approval ticket or on paper. If an attacker can alter the files on the media they can alter a manifest sitting next to them.

Keep the transfer set small and single-purpose. One import per ticket makes it clear what changed when something misbehaves, and makes rollback a matter of deleting a folder.

Copy the transfer set to the isolated server and run the checksum check first. Do not unpack, load or install anything that fails it. sha256sum -c prints a line per file and a final warning if any file differs, so a clean run is silent apart from the OK lines.

Load the image archives with docker load. The Docker documentation shows both docker load < file.tar.gz and docker load --input file.tar, and it prints Loaded image: with the name and tag when it succeeds. Then install packages from the wheelhouse with the index off.

If you will use Ollama, import the GGUF with a Modelfile. The Ollama documentation says a single GGUF file needs a Modelfile with a FROM line pointing at the file, then ollama create and ollama run. It also says Ollama does not quantise a GGUF during import, so the quantisation you downloaded is the one you get.

Libraries built around the Hugging Face Hub try to reach it by default, even when the files are local. Set HF_HUB_OFFLINE=1. The documentation says that with it set, no HTTP calls are made to the Hub and only cached files are accessed, and that if a file is not in the cache an error is raised. That error is useful: it tells you what you forgot to stage.

For vLLM, the troubleshooting documentation recommends downloading the model first with the hf CLI and passing the local path to vLLM. Mount the model directory into the container and point --model at the mounted path. vLLM also collects usage statistics by default. Its documentation gives two ways to turn that off: VLLM_NO_USAGE_STATS=1 or DO_NOT_TRACK=1, or creating a do_not_track file under ~/.config/vllm. In an isolated network the call would fail anyway, but a deliberate setting is better than a failure someone has to explain.

Bind the server to an internal address only, and require an API key as you would on a connected network. How to self-host an LLM covers the memory sizing, authentication and TLS in detail.

Believing the network is closed is not the same as showing it. Run two checks and keep the output. First, on the host, list established TCP connections with the processes that own them. ss -t state established lists established TCP sockets, -n stops it resolving names, and -p adds the process. Run it while the model is serving requests and during a restart, and look for anything that is not your own clients.

Second, check at the edge. The firewall or switch between the server and the rest of the network should deny outbound traffic by default and log denied attempts. A log with zero denied outbound attempts from the model server across a week of use is stronger evidence than a one-off look at a terminal. If you do see attempts, they tell you which component is still trying to reach the internet, and you can fix the setting rather than rely on the firewall.

For an extra isolation layer on a single container that does not need to serve other machines, Docker can start it with --network none, which creates only a loopback device. That suits batch jobs and test runs. A server that other machines must reach needs a network, so there you rely on the host and edge checks.

Decide how often you will import. A quiet air-gapped environment quietly falls behind: models, container images, drivers and operating system patches all age. Pick a cadence, such as monthly for patches and per request for new models, and make someone responsible for running it.

Repeat steps 2 to 6 for every update, with a new ticket and a new manifest. Keep the previous model directory and image until the new one has passed your acceptance prompts, then remove it. Run a fixed set of test prompts before and after each import so you can see behaviour changes, not just that the process starts. How to validate your AI explains how to build that set.

Log requests and responses inside the gap according to your data rules, and decide how logs are retained and who can read them. If nothing may leave, review logs inside the gap. If a reviewed summary may leave, agree its format in advance.