Imported from NVIDIA/deepops (skills/deploy-airgapped/SKILL.md). Install upstream with npx skills add NVIDIA/deepops --skill deploy-airgapped. Copyright stays with the author.

Deploy DeepOps in an air-gapped environment

Preconditions and boundaries

• Use an Internet-connected staging machine and a separate provisioning machine that can reach every target and every internal mirror. Do not connect an isolated network to the Internet for this procedure.
• Fix one DeepOps commit, target OS release, CPU architecture, cluster type, inventory, and enabled feature set before mirroring. Repository and image requirements change when any of these inputs change.
• Provide complete distribution package mirrors plus internal HTTP package/file service and container registry endpoints. Use trusted TLS where possible; if the registry is intentionally insecure, configure it explicitly.
• Bootstrap the isolated provisioning machine with the OS packages and Python environment required by scripts/setup.sh before closing the boundary, or provide approved internal OS and Python package indexes. Copying the source tree alone does not install Ansible.
• Reserve enough storage for repository metadata, packages, container images, charts, the initialized DeepOps checkout, and two copies during transfer.
• Confirm that every target is fully manageable and has no active users or workloads. Driver installation can reboot hosts.
• Run DeepOps commands from the repository root. Keep secrets out of archives and store site configuration in config/, never config.example/.

DeepOps does not provide supported mirror-building automation. Do not run scripts/airgap/build_offline_cache.sh: it calls the removed playbooks/airgap/build-offline-cache.yml and cannot complete in this tree. Build and verify the mirrors explicitly as described below.

1. Freeze the deployment inputs

1. Record the DeepOps commit and initialize every submodule on the connected staging machine:

   git rev-parse HEAD
   git submodule update --init --recursive
   ./scripts/setup.sh

2. Copy config.example/ to config/. Define the real hosts in config/inventory before collecting artifacts so the selected cluster path and optional components are known:

   • Slurm: controller/login hosts in [slurm-master], compute hosts in [slurm-node]; a single host may be in both groups.
   • Kubernetes: control-plane hosts in [kube_control_plane] and [etcd], workers in [kube_node]; a single host may be in all three groups.

3. Review the resolved inventory. Stop if it contains any unintended host:

   ansible-inventory -i config/inventory --list

4. Record the target OS release/architecture and all enabled roles. Mirror every dependency of that exact profile. In particular, account for:

   • distribution, Docker CE, NVIDIA CUDA/driver, NVIDIA Container Toolkit, and EPEL repositories where applicable;
   • direct-download archives referenced by enabled role defaults;
   • every container image and Helm chart used by the chosen Slurm or Kubernetes path;
   • the initialized Kubespray submodule, Galaxy roles/collections, and Python packages needed on the provisioning machine.

2. Build package mirrors on the connected side

Ubuntu/APT

Install apt-mirror, set base_path in /etc/apt/mirror.list (for example, /var/repos), and add the suites for the target release. For Ubuntu 24.04 (noble), the current DeepOps documentation uses these sources:

deb http://archive.ubuntu.com/ubuntu noble main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu noble-security main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu noble-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu noble-backports main restricted universe multiverse
deb https://download.docker.com/linux/ubuntu noble stable
deb https://nvidia.github.io/libnvidia-container/stable/deb/amd64 /
deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64 /

Add DGX OS or MAAS repositories only when the selected profile uses them. Use the matching suite and architecture instead of noble/amd64 for another target. Download exactly the configured set:

sudo mkdir -p /var/repos
sudo apt-mirror

Download the signing material that DeepOps fetches directly, plus the RPM bootstrap files when building an Enterprise Linux mirror. Keep these outside mirror/ so their archive paths are deterministic:

sudo mkdir -p /var/repos/keys /var/repos/bootstrap
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /var/repos/keys/docker-ubuntu.asc
sudo curl -fsSL https://download.docker.com/linux/centos/gpg \
  -o /var/repos/keys/docker-rpm.asc
sudo curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  -o /var/repos/keys/libnvidia-container.asc
sudo curl -fsSL \
  https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/3bf863cc.pub \
  -o /var/repos/keys/cuda-ubuntu2404-3bf863cc.pub
sudo curl -fsSL \
  https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/D42D0685.pub \
  -o /var/repos/keys/cuda-rhel9-D42D0685.pub
sudo curl -fsSL \
  https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-9 \
  -o /var/repos/keys/RPM-GPG-KEY-EPEL-9
sudo curl -fsSL \
  https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm \
  -o /var/repos/bootstrap/epel-release-latest-9.noarch.rpm
gpg --show-keys --with-fingerprint /var/repos/keys/*

Replace ubuntu2404, rhel9, and 9 together for another supported target. Verify the downloaded fingerprints against the vendor's published values under the site's package-trust process before transfer; do not turn off GPG checking.

Enterprise Linux/RPM

Configure the target distribution repositories plus EPEL, Docker CE, CUDA, libnvidia-container, and nvidia-container-toolkit. For EL9, download the checked-in documentation's complete additional-repository set into the top-level repo-ID directories created by reposync:

for repo_id in epel docker-ce-stable cuda-rhel9-x86_64 \
  libnvidia-container nvidia-container-toolkit; do
  sudo reposync -l --repoid="${repo_id}" --downloadcomps \
    --download-metadata --download_path=/var/repos
  sudo createrepo "/var/repos/${repo_id}"
done

Use cuda-rhel8-x86_64 instead on EL8. Compare the final IDs and package contents with /etc/yum.repos.d/ and the enabled workload on a connected reference host; distribution repositories remain a separate prerequisite.

3. Mirror direct files, charts, and images

1. Download the direct files selected by the current default Slurm profile. These names and versions come from the checked-in role defaults:

   sudo install -d -m 0755 /var/repos/downloads
   while read -r url name; do
     sudo curl -fL "${url}" -o "/var/repos/downloads/${name}"
   done <<'EOF'
   https://download.schedmd.com/slurm/slurm-26.05.1.tar.bz2 slurm-26.05.1.tar.bz2
   https://download.open-mpi.org/release/hwloc/v2.5/hwloc-2.5.0.tar.gz hwloc-2.5.0.tar.gz
   https://github.com/openpmix/openpmix/releases/download/v3.2.3/pmix-3.2.3.tar.bz2 pmix-3.2.3.tar.bz2
   https://github.com/mej/nhc/releases/download/1.4.3/lbnl-nhc-1.4.3.tar.xz lbnl-nhc-1.4.3.tar.xz
   https://github.com/NVIDIA/pyxis/archive/v0.11.1.tar.gz pyxis-0.11.1.tar.gz
   https://developer.download.nvidia.com/hpc-sdk/23.7/nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz
   https://get.helm.sh/helm-v3.17.1-linux-amd64.tar.gz helm-v3.17.1-linux-amd64.tar.gz
   EOF
   for name in slurm-26.05.1.tar.bz2 hwloc-2.5.0.tar.gz \
     pmix-3.2.3.tar.bz2 lbnl-nhc-1.4.3.tar.xz pyxis-0.11.1.tar.gz \
     nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz \
     helm-v3.17.1-linux-amd64.tar.gz; do
     test -s "/var/repos/downloads/${name}"
   done

   Enroot is also enabled by default, but its package filenames are owned by the pinned nvidia.enroot Galaxy role rather than this repository. After ./scripts/setup.sh, use that installed role's real package list to download the target-OS packages and generate the matching Ansible override. Choose enroot_deb_packages for Ubuntu or enroot_rpm_packages for Enterprise Linux; repeat for both when the inventory mixes OS families:

   test -s roles/galaxy/nvidia.enroot/defaults/main.yml
   cat >/tmp/deepops-mirror-enroot.yml <<'EOF'
   ---
   - hosts: localhost
     connection: local
     gather_facts: true
     become: true
     vars_files:
       - "{{ deepops_root }}/roles/galaxy/nvidia.enroot/defaults/main.yml"
     tasks:
       - name: Download the pinned Enroot packages
         get_url:
           url: "{{ item }}"
           dest: "/var/repos/downloads/{{ item | basename }}"
           mode: "0644"
         loop: "{{ lookup('vars', enroot_package_list_var) }}"
       - name: Write the matching offline override
         copy:
           dest: "/var/repos/downloads/{{ enroot_package_list_var }}-offline.yml"
           mode: "0644"
           content: |
             {{ enroot_package_list_var }}:
             {% for package_url in lookup('vars', enroot_package_list_var) %}
               - "{{ offline_http_base }}/{{ package_url | basename }}"
             {% endfor %}
   EOF
   ansible-playbook -i localhost, /tmp/deepops-mirror-enroot.yml \
     -e "deepops_root=$(pwd)" \
     -e enroot_package_list_var=enroot_deb_packages \
     -e offline_http_base=http://package-server/downloads

   Replace package-server before running the generator. Verify every file is nonempty. Use these exact overrides with the published paths:

   slurm_src_url: "http://package-server/downloads/slurm-26.05.1.tar.bz2"
   hwloc_src_url: "http://package-server/downloads/hwloc-2.5.0.tar.gz"
   pmix_src_url: "http://package-server/downloads/pmix-3.2.3.tar.bz2"
   nhc_src_url: "http://package-server/downloads/lbnl-nhc-1.4.3.tar.xz"
   slurm_pyxis_tarball_url: "http://package-server/downloads/pyxis-0.11.1.tar.gz"
   hpcsdk_download_url: "http://package-server/downloads/nvhpc_2023_237_Linux_x86_64_cuda_12.2.tar.gz"

   Add the generated Enroot YAML list to config/group_vars/all.yml. On Enterprise Linux, also override epel_package and epel_key_url; several default roles import the EPEL key directly, so mirroring packages alone is insufficient.

   DCGM needs no separate file mirror: the nvidia_dcgm role installs dcgm_pkg_name (datacenter-gpu-manager) from the CUDA package repository, so it is covered by the CUDA repository mirror below — verify the DCGM package is present in the mirrored CUDA repo content.

   The default Slurm configuration also enables CUDA installation, whose Ubuntu and Enterprise Linux tasks fetch the CUDA repository and its signing material from developer.download.nvidia.com. Mirroring the CUDA packages alone does not redirect these tasks. On Ubuntu, the role uses nvidia_driver_ubuntu_cuda_keyring_url only to install the keyring package; it does not use nvidia_driver_ubuntu_cuda_repo_baseurl to write an APT source. The keyring package writes an upstream source that must be replaced on every target as shown in step 5. On Enterprise Linux, override both repository variables after mirroring the metadata and GPG key:

   • nvidia_driver_ubuntu_cuda_keyring_url (Ubuntu, plus the target-side APT source replacement in step 5)
   • nvidia_driver_rhel_cuda_repo_baseurl and nvidia_driver_rhel_cuda_repo_gpgkey (Enterprise Linux)

   Either mirror and override every enabled dependency or explicitly disable the optional component. Do not silently fall back to its public URL.

2. Mirror the Helm repositories/charts required by the selected Kubernetes profile. The default tree references the stable Helm repository, GPU Operator v26.3.3, and, when enabled, NFS subdir external provisioner 4.0.18. Set gpu_operator_helm_repo and k8s_nfs_client_helm_repo to internal chart repositories.

   Archive the chart versions selected by current DeepOps defaults on the connected side:

   mkdir -p /tmp/charts
   helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
   helm pull nvidia/gpu-operator --version v26.3.3 --destination /tmp/charts
   helm repo add nfs-subdir-external-provisioner \
     https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
   helm pull nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
     --version 4.0.18 --destination /tmp/charts
   helm repo index /tmp/charts

   Omit the NFS chart only when k8s_nfs_client_provisioner: false. Publish /tmp/charts from an internal HTTP service and set the two repository variables to that URL.

3. Generate and collect the exact Kubespray file and image set from the pinned submodule. generate_list.yml runs only against localhost; passing -i config/inventory alone does not apply variables from DeepOps' k8s_cluster group. Export the effective variables for one representative Kubernetes host, review the selected versions, and pass that JSON as extra-vars so the localhost play uses the deployment's resolved overrides. Kubernetes download versions must be cluster-wide; stop if control-plane and worker hosts resolve different artifact-selection variables. Then render the enabled DeepOps charts with the same values used by their roles and append their images to Kubespray's list before creating the image archive:

   task_deepops_root=$(pwd)
   task_offline_dir="${task_deepops_root}/submodules/kubespray/contrib/offline"
   task_kube_host=kube01 # replace with one real kube_control_plane or kube_node host
   ansible-inventory -i config/inventory --host "${task_kube_host}" \
     > /tmp/kubespray-effective-vars.json
   python3 - /tmp/kubespray-effective-vars.json <<'PY'
   import json
   import sys
   
   variables = json.load(open(sys.argv[1], encoding="utf-8"))
   for name in ("kube_version", "container_manager", "etcd_deployment_type"):
       print(f"{name}={variables.get(name, '<Kubespray default>')}")
   PY
   cd "${task_offline_dir}"
   ./generate_list.sh -i "${task_deepops_root}/config/inventory" \
     -e "@/tmp/kubespray-effective-vars.json"
   test -s temp/files.list
   test -s temp/images.list
   NO_HTTP_SERVER=1 ./manage-offline-files.sh
   
   helm template nvidia-gpu-operator \
     /tmp/charts/gpu-operator-v26.3.3.tgz \
     --namespace gpu-operator \
     --set driver.version=580.126.20 \
     --set mig.strategy=mixed \
     --set driver.enabled=false \
     --set toolkit.enabled=true \
     --set dcgm.enabled=false \
     --set migManager.enabled=true \
     > /tmp/gpu-operator-rendered.yml
   helm template nfs-subdir-external-provisioner \
     /tmp/charts/nfs-subdir-external-provisioner-4.0.18.tgz \
     > /tmp/nfs-provisioner-rendered.yml
   python3 - /tmp/gpu-operator-rendered.yml \
     /tmp/nfs-provisioner-rendered.yml <<'PY' >> temp/images.list
   import sys
   import yaml
   
   images = set()
   
   def walk(value):
       if isinstance(value, dict):
           repository = value.get("repository")
           image = value.get("image")
           version = value.get("version", value.get("tag"))
           if isinstance(repository, str) and isinstance(image, str) and version:
               reference = f"{repository.rstrip('/')}/{image.lstrip('/')}"
               if "@" not in reference and ":" not in reference.rsplit("/", 1)[-1]:
                   separator = "@" if str(version).startswith("sha256:") else ":"
                   reference = f"{reference}{separator}{version}"
               images.add(reference)
           if isinstance(image, str):
               tail = image.rsplit("/", 1)[-1]
               if "@" in image or ":" in tail:
                   images.add(image)
           for child in value.values():
               walk(child)
       elif isinstance(value, list):
           for child in value:
               walk(child)
   
   for manifest_path in sys.argv[1:]:
       with open(manifest_path, encoding="utf-8") as stream:
           for document in yaml.safe_load_all(stream):
               walk(document)
   
   if not images:
       raise SystemExit("no chart images found")
   print("\n".join(sorted(images)))
   PY
   sort -u -o temp/images.list temp/images.list
   test -z "$(grep -Ev \
     '^[^[:space:]]+(:[^/[:space:]]+|@sha256:[0-9a-f]+)$' \
     temp/images.list)"
   
   PRIVATE_REGISTRY=nvcr.io IMAGES_FROM_FILE="${task_offline_dir}/temp/images.list" \
     ./manage-offline-container-images.sh create
   install -d /tmp/kubespray-offline
   cp offline-files.tar.gz container-images.tar.gz temp/files.list \
     temp/images.list /tmp/kubespray-offline/
   cd "${task_deepops_root}"

   The PRIVATE_REGISTRY=nvcr.io setting makes the checked-in image manager strip nvcr.io/ just as it strips the Kubespray source registries, so an internal registry mirror can serve the paths that the GPU Operator chart requests. If NFS is disabled, omit its helm template command. If site overrides change any GPU Operator flag, render with those values instead. The inventory path is still required by the generator, but the explicit effective-variable JSON is what makes group/host version overrides visible to its localhost play. Treat a missing representative host, an empty list, a failed pull, or an unparseable rendered image as a collection failure; do not continue with a partial archive.

4. Pull and archive the CUDA validator plus the images used by enabled Slurm monitoring/registry roles. For the current validator and registry images:

   mkdir -p /tmp/images
   docker pull nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
   docker save -o /tmp/images/nvidia-cuda-12.4.1-base-ubuntu22.04.tar \
     nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04
   docker pull registry:3.1.1
   docker save -o /tmp/images/registry-3.1.1.tar registry:3.1.1

   When Slurm monitoring stays enabled, also mirror the images selected by prometheus_container, grafana_container, alertmanager_container, node_exporter_container, and nvidia_dcgm_container, then override those variables with their internal registry paths. The Slurm exporter is a special case: its default slurm_exporter_build_image: true builds from golang:1.24 and ubuntu:24.04, and the runtime stage runs apt-get. Build it while connected, archive it with the other images, and use the prebuilt image offline:

   for image in prom/prometheus:v3.13.0 grafana/grafana:13.1.0 \
     prom/alertmanager:v0.33.0 quay.io/prometheus/node-exporter:v1.11.1 \
     nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless; do
     docker pull "${image}"
   done
   docker save -o /tmp/images/slurm-monitoring.tar \
     prom/prometheus:v3.13.0 grafana/grafana:13.1.0 \
     prom/alertmanager:v0.33.0 quay.io/prometheus/node-exporter:v1.11.1 \
     nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless
   docker build -t deepops/prometheus-slurm-exporter:2.0.0 \
     roles/prometheus-slurm-exporter/files/docker
   docker save -o /tmp/images/prometheus-slurm-exporter-2.0.0.tar \
     deepops/prometheus-slurm-exporter:2.0.0

   This connected-side build obtains the two base images and runtime APT packages. Do not leave the default local build enabled in the isolated environment unless an approved internal build setup supplies both base images and APT sources.

5. Archive the initialized checkout separately from site secrets. Keep .git in the archive: playbooks/k8s-cluster.yml unconditionally runs git submodule update --init from the repository root, so an extracted tree without Git metadata fails before Kubespray starts. With submodules already initialized at the pinned commits, that task is an offline no-op. Run this from the checkout's parent directory after setup.sh has populated its Galaxy dependencies and git submodule update --init --recursive has completed:

   git -C deepops submodule status --recursive   # every line must start with a space (initialized, pinned)
   tar --exclude='deepops/config' \
     -czf /tmp/deepops-source.tar.gz deepops

   The archive is larger with Git metadata included; that is the price of the Kubernetes path working. Excluding deepops/config still keeps site secrets out of the transfer artifact.

6. Package repositories and images for approved removable-media transfer and create checksums. Substitute a protected, user-owned staging directory for /path/to/staging; create the ISO files without elevated privileges:

   genisoimage -o /path/to/staging/packages.iso /var/repos
   genisoimage -o /path/to/staging/images.iso /tmp/images
   genisoimage -o /path/to/staging/charts.iso /tmp/charts
   genisoimage -o /path/to/staging/kubespray-offline.iso /tmp/kubespray-offline
   genisoimage -o /path/to/staging/keys.iso /var/repos/keys
   cp /tmp/deepops-source.tar.gz /path/to/staging/
   cd /path/to/staging
   sha256sum packages.iso images.iso charts.iso kubespray-offline.iso keys.iso \
     deepops-source.tar.gz > SHA256SUMS

Transfer the ISO files, checkout archive, and checksum file through the site's approved boundary process. Verify the checksums again inside the isolated network before importing anything:

cd /path/to/import
sha256sum -c SHA256SUMS

4. Import artifacts on the isolated side

1. Mount the verified transfer images and copy their contents to persistent or working directories:

   sudo mkdir -p /mnt/deepops-packages /mnt/deepops-images \
     /mnt/deepops-charts /mnt/deepops-kubespray /mnt/deepops-keys
   sudo mount -o loop /path/to/import/packages.iso /mnt/deepops-packages
   sudo mount -o loop /path/to/import/images.iso /mnt/deepops-images
   sudo mount -o loop /path/to/import/charts.iso /mnt/deepops-charts
   sudo mount -o loop /path/to/import/kubespray-offline.iso /mnt/deepops-kubespray
   sudo mount -o loop /path/to/import/keys.iso /mnt/deepops-keys
   sudo mkdir -p /var/repos /var/www/html/charts
   mkdir -p /tmp/images
   sudo cp -a /mnt/deepops-packages/. /var/repos/
   cp -a /mnt/deepops-images/. /tmp/images/
   sudo cp -a /mnt/deepops-charts/. /var/www/html/charts/
   mkdir -p /tmp/kubespray-offline
   cp -a /mnt/deepops-kubespray/. /tmp/kubespray-offline/

2. Publish signing keys first, and fail if any expected file is missing. This EL9/Ubuntu 24.04 example corresponds exactly to the connected-side commands:

   sudo install -d /var/www/html/keys
   for key in docker-ubuntu.asc docker-rpm.asc libnvidia-container.asc \
     cuda-ubuntu2404-3bf863cc.pub cuda-rhel9-D42D0685.pub \
     RPM-GPG-KEY-EPEL-9; do
     test -s "/mnt/deepops-keys/${key}"
     sudo install -m 0644 "/mnt/deepops-keys/${key}" "/var/www/html/keys/${key}"
   done
   gpg --show-keys --with-fingerprint /var/www/html/keys/*
   sha256sum /var/www/html/keys/*

   For APT, preserve apt-mirror's hostname and path layout verbatim:

   sudo install -d /var/www/html/repos
   test -d /var/repos/mirror/archive.ubuntu.com/ubuntu/dists/noble
   test -d /var/repos/mirror/download.docker.com/linux/ubuntu/dists/noble
   test -d /var/repos/mirror/nvidia.github.io/libnvidia-container/stable/deb/amd64
   test -d /var/repos/mirror/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64
   sudo cp -a /var/repos/mirror/. /var/www/html/repos/

   Publish the direct Slurm/Enroot files under the paths used by the generated overrides:

   test -d /var/repos/downloads
   sudo install -d /var/www/html/downloads
   sudo cp -a /var/repos/downloads/. /var/www/html/downloads/
   test -z "$(find /var/www/html/downloads -maxdepth 1 -type f -size 0 -print -quit)"

   For RPM, publish the exact top-level repo-ID directories, EPEL bootstrap RPM, and a rewritten Container Toolkit repo file whose base URLs are internal:

   for repo_id in epel docker-ce-stable cuda-rhel9-x86_64 \
     libnvidia-container nvidia-container-toolkit; do
     test -d "/var/repos/${repo_id}/repodata"
     sudo cp -a "/var/repos/${repo_id}" /var/www/html/repos/
   done
   test -s /var/repos/bootstrap/epel-release-latest-9.noarch.rpm
   sudo install -m 0644 /var/repos/bootstrap/epel-release-latest-9.noarch.rpm \
     /var/www/html/repos/epel-release-latest-9.noarch.rpm
   sudo tee /var/www/html/repos/nvidia-container-toolkit.repo >/dev/null <<'EOF'
   [libnvidia-container]
   name=libnvidia-container
   baseurl=http://package-server/repos/libnvidia-container
   enabled=1
   gpgcheck=1
   gpgkey=http://package-server/keys/libnvidia-container.asc
   
   [nvidia-container-toolkit]
   name=nvidia-container-toolkit
   baseurl=http://package-server/repos/nvidia-container-toolkit
   enabled=1
   gpgcheck=1
   gpgkey=http://package-server/keys/libnvidia-container.asc
   EOF

   Replace package-server with the real internal DNS name in both the repo file and step 5. Verify every configured file URL with curl -fsS and every APT/RPM metadata path with the target package manager before deployment. Preserve GPG verification.

3. Load the registry image and start the isolated registry if the site does not already provide one. Run this fallback on the registry host itself. It is plain HTTP, so the imports below push through Docker's IPv4 loopback exception (127.0.0.1:5000); cluster nodes use the reachable registry-host:5000 name configured later. Do not push to registry-host:5000 from the importer unless that Docker daemon has been separately and deliberately configured for the insecure endpoint:

   docker load -i /tmp/images/registry-3.1.1.tar
   docker volume create registry-images
   docker run -d -p 5000:5000 --restart=always --name registry \
     -v registry-images:/var/lib/registry registry:3.1.1

4. Import, retag, and push every archived image on that registry host. Preserve the complete repository path expected by the consuming chart or role. The hostname used for the push is not part of the stored repository path, so a cluster pull of registry-host:5000/nvidia/cuda:... resolves the image pushed as 127.0.0.1:5000/nvidia/cuda:...:

   docker load -i /tmp/images/nvidia-cuda-12.4.1-base-ubuntu22.04.tar
   docker tag nvcr.io/nvidia/cuda:12.4.1-base-ubuntu22.04 \
     127.0.0.1:5000/nvidia/cuda:12.4.1-base-ubuntu22.04
   docker push 127.0.0.1:5000/nvidia/cuda:12.4.1-base-ubuntu22.04
   docker load -i /tmp/images/prometheus-slurm-exporter-2.0.0.tar
   docker tag deepops/prometheus-slurm-exporter:2.0.0 \
     127.0.0.1:5000/deepops/prometheus-slurm-exporter:2.0.0
   docker push 127.0.0.1:5000/deepops/prometheus-slurm-exporter:2.0.0
   docker load -i /tmp/images/slurm-monitoring.tar
   while read -r source target; do
     docker tag "${source}" "${target}"
     docker push "${target}"
   done <<'EOF'
   prom/prometheus:v3.13.0 127.0.0.1:5000/prom/prometheus:v3.13.0
   grafana/grafana:13.1.0 127.0.0.1:5000/grafana/grafana:13.1.0
   prom/alertmanager:v0.33.0 127.0.0.1:5000/prom/alertmanager:v0.33.0
   quay.io/prometheus/node-exporter:v1.11.1 127.0.0.1:5000/prometheus/node-exporter:v1.11.1
   nvcr.io/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless 127.0.0.1:5000/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless
   EOF

5. Extract the initialized DeepOps checkout on the provisioning machine:

   mkdir -p "$HOME/deepops-src"
   tar -xzf /path/to/import/deepops-source.tar.gz -C "$HOME/deepops-src"
   cd "$HOME/deepops-src/deepops"

   Activate the provisioning environment prepared before isolation (or install it from approved internal indexes), then copy and edit config.example/ if site configuration was not transferred separately. Install the mirrored Helm version expected by scripts/k8s/install_helm.sh on an amd64 Linux provisioning host so that script does not fetch its public installer:

   mkdir -p /tmp/deepops-helm
   tar -xzf /var/repos/downloads/helm-v3.17.1-linux-amd64.tar.gz \
     -C /tmp/deepops-helm
   sudo install -m 0755 /tmp/deepops-helm/linux-amd64/helm /usr/local/bin/helm
   helm version --short | grep '^v3.17.1'

   Mirror the matching archive for another provisioning platform; do not use the amd64 archive there.

6. Publish the Kubespray static-file tree, then import every Kubespray, GPU Operator, and NFS image collected in step 3 into the registry. Do not run the pinned manager's register action: before importing, it overwrites /etc/docker/daemon.json or /etc/containers/registries.conf, substitutes the importer's hostname instead of DESTINATION_REGISTRY, and does not back up, restore, or restart the runtime. Extract its archive and perform the same load/tag/push loop without changing host configuration:

   sudo install -d /var/www/html/kubespray
   sudo tar -xzf /tmp/kubespray-offline/offline-files.tar.gz \
     -C /var/www/html/kubespray
   test -d /var/www/html/kubespray/offline-files
   
   task_kubespray_import=$(mktemp -d)
   (
     set -euo pipefail
     trap 'rm -rf "${task_kubespray_import}"' EXIT
     tar -xzf /tmp/kubespray-offline/container-images.tar.gz \
       -C "${task_kubespray_import}"
     while read -r archive repository; do
       test -n "${archive}" && test -n "${repository}"
       load_line=$(docker load \
         -i "${task_kubespray_import}/container-images/${archive}" | sed -n '1p')
       source_ref=$(printf '%s\n' "${load_line}" | awk '{print $3}')
       if [ "${source_ref}" = ID: ]; then
         source_ref=$(printf '%s\n' "${load_line}" | awk '{print $4}')
       fi
       image_id=$(docker image inspect --format '{{.Id}}' "${source_ref}")
       test -n "${image_id}"
       docker tag "${image_id}" "127.0.0.1:5000/${repository}"
       docker push "127.0.0.1:5000/${repository}"
     done < "${task_kubespray_import}/container-images/container-images.txt"
   )

   These commands assume the fallback plain-HTTP registry is local to the importer. For a site registry on another host, require trusted TLS or have the site administrator configure and restart the importer's runtime under site change control before substituting that registry endpoint. Confirm the imported repositories through the registry's normal catalog or pull checks before deployment.

5. Configure DeepOps for internal endpoints

Set mirror overrides in config/group_vars/all.yml (or a narrower group file). Use values matching the paths actually published by the site:

docker_ubuntu_repo_base_url: "http://package-server/repos/download.docker.com/linux/ubuntu"
docker_ubuntu_repo_gpgkey: "http://package-server/keys/docker-ubuntu.asc"
nvidia_container_toolkit_repo_base_url: "http://package-server/repos/nvidia.github.io/libnvidia-container"
nvidia_container_toolkit_repo_gpg_url: "http://package-server/keys/libnvidia-container.asc"

docker_rh_repo_base_url: "http://package-server/repos/docker-ce-stable"
docker_rh_repo_gpgkey: "http://package-server/keys/docker-rpm.asc"
nvidia_container_toolkit_rpm_repo_url: "http://package-server/repos/nvidia-container-toolkit.repo"

docker_insecure_registries:
  - "registry-host:5000"

# CUDA repository mirror (required by the default Slurm path; see below)
nvidia_driver_ubuntu_cuda_keyring_url: "http://package-server/repos/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb"
nvidia_driver_rhel_cuda_repo_baseurl: "http://package-server/repos/cuda-rhel9-x86_64/"
nvidia_driver_rhel_cuda_repo_gpgkey: "http://package-server/keys/cuda-rhel9-D42D0685.pub"

# EPEL mirror (Enterprise Linux; key import fails offline without this)
epel_package: "http://package-server/repos/epel-release-latest-9.noarch.rpm"
epel_key_url: "http://package-server/keys/RPM-GPG-KEY-EPEL-9"

For Ubuntu 24.04, prepare every target before running the Slurm playbook or any play that includes nvidia_cuda or nvidia_dcgm. Replace package-server with the real internal host first. These commands install the mirrored keyring, remove the public source that package creates, write the internal cuda-compute-repo.list, update that source in isolation, and then prove the complete APT update resolves no public NVIDIA host before allowing a normal update:

(
  set -euo pipefail
  task_cuda_repo_url='http://package-server/repos/developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64'
  task_cuda_keyring_url="${task_cuda_repo_url}/cuda-keyring_1.1-1_all.deb"

  case "${task_cuda_repo_url}" in
    *package-server*|https://developer.download.nvidia.com/*|http://developer.download.nvidia.com/*)
      echo 'ERROR: set task_cuda_repo_url to the internal CUDA mirror' >&2
      exit 1
      ;;
  esac

  task_cuda_keyring_deb=$(mktemp)
  task_apt_uris=$(mktemp)
  trap 'rm -f "${task_cuda_keyring_deb}" "${task_apt_uris}"' EXIT
  curl -fsSL "${task_cuda_keyring_url}" -o "${task_cuda_keyring_deb}"
  test -s "${task_cuda_keyring_deb}"
  sudo dpkg -i "${task_cuda_keyring_deb}"

  if [ -e /etc/apt/sources.list.d/cuda-ubuntu2404-x86_64.list ]; then
    sudo mv --backup=numbered -- \
      /etc/apt/sources.list.d/cuda-ubuntu2404-x86_64.list \
      /etc/apt/sources.list.d/cuda-ubuntu2404-x86_64.list.disabled
  fi
  printf 'deb [signed-by=/usr/share/keyrings/cuda-archive-keyring.gpg] %s /\n' \
    "${task_cuda_repo_url%/}" |
    sudo tee /etc/apt/sources.list.d/cuda-compute-repo.list >/dev/null

  sudo apt-get \
    -o Dir::Etc::sourcelist='sources.list.d/cuda-compute-repo.list' \
    -o Dir::Etc::sourceparts='-' update

  apt-get --print-uris update | tee "${task_apt_uris}"
  task_public_nvidia_urls=$(
    sed -n "s/^'\(https\?:\/\/[^']*\)'.*/\1/p" "${task_apt_uris}" |
      awk -F/ '{ host = tolower($3); sub(/:.*/, "", host) }
                host == "developer.download.nvidia.com" ||
                host == "repo.download.nvidia.com" ||
                host == "nvidia.github.io" { print }'
  )
  if [ -n "${task_public_nvidia_urls}" ]; then
    printf 'ERROR: public NVIDIA APT URL still resolves:\n%s\n' \
      "${task_public_nvidia_urls}" >&2
    exit 1
  fi
  sudo apt-get update
)

Use the matching repository directory and keyring-generated source filename on another Ubuntu release. Keep all other APT sources internal too: the final apt-get update intentionally checks the complete target configuration, which is what the role will update. Retain the nvidia_driver_ubuntu_cuda_keyring_url override so the role finds the mirrored package; do not set nvidia_driver_ubuntu_cuda_repo_baseurl expecting it to redirect APT.

For Slurm, disable the default pull-through registry cache when upstream Docker Hub is unreachable. If using the site's registry, the smallest honest profile is:

slurm_enable_container_registry: false
standalone_container_registry_cache_enable: false
slurm_exporter_build_image: false
slurm_exporter_container: "registry-host:5000/deepops/prometheus-slurm-exporter:2.0.0"
prometheus_container: "registry-host:5000/prom/prometheus:v3.13.0"
grafana_container: "registry-host:5000/grafana/grafana:13.1.0"
alertmanager_container: "registry-host:5000/prom/alertmanager:v0.33.0"
node_exporter_container: "registry-host:5000/prometheus/node-exporter:v1.11.1"
nvidia_dcgm_container: "registry-host:5000/nvidia/k8s/dcgm-exporter:4.5.3-4.8.2-distroless"

Mirror and override the Slurm source/archive variables listed above. If their artifacts were deliberately excluded, disable the corresponding optional features, for example slurm_enable_monitoring, slurm_install_hpcsdk, slurm_install_nhc, slurm_install_enroot, or slurm_install_pyxis. Do not disable a required feature merely to make a playbook pass.

For Kubernetes, add the pinned Kubespray offline rewrites and the DeepOps chart repositories. This layout matches the static-file extraction and image-manager registration commands above:

registry_host: "registry-host:5000"
registry_addr: "registry-host:5000"
files_repo: "http://package-server/kubespray/offline-files"

kube_image_repo: "{{ registry_host }}"
gcr_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"
github_image_repo: "{{ registry_host }}"
github_url: "{{ files_repo }}/github.com"
dl_k8s_io_url: "{{ files_repo }}/dl.k8s.io"
storage_googleapis_url: "{{ files_repo }}/storage.googleapis.com"
get_helm_url: "{{ files_repo }}/get.helm.sh"
local_path_provisioner_helper_image_repo: "{{ registry_host }}/busybox"

gpu_operator_helm_repo: "http://package-server/charts"
k8s_nfs_client_helm_repo: "http://package-server/charts"

containerd_registries_mirrors:
  - prefix: "{{ registry_addr }}"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: "nvcr.io"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: "registry.k8s.io"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: "docker.io"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: "quay.io"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: "gcr.io"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true
  - prefix: "ghcr.io"
    mirrors:
      - host: "http://{{ registry_addr }}"
        capabilities: ["pull", "resolve"]
        skip_verify: true

Use trusted TLS and skip_verify: false when the internal registry provides it. Set k8s_nfs_client_provisioner: false only if the cluster uses a site-owned storage path or intentionally has no dynamic NFS provisioner.

The current top-level Kubernetes playbook invokes a Helm installer URL and adds https://charts.helm.sh/stable; when the Ansible host platform differs from the cluster nodes (for example a macOS or arm64 control machine), its artifact step also downloads kubectl and its checksum directly from https://dl.k8s.io. Run the deployment from a control host matching the cluster platform so the playbook fetches kubectl from a cluster node instead, or pre-approve an internal mirror for that URL. Its containerd local-registry settings also carry an explicit TODO in config.example/group_vars/k8s_cluster.yml. Therefore, before declaring a fully disconnected Kubernetes run ready, prove that these references are satisfied by approved internal endpoints or obtain a DeepOps code change. Do not claim that the current playbook is turnkey offline.

6. Preflight and deploy

Run the machine-readable doctor from the isolated provisioning machine:

python3 scripts/validation/deepops_doctor.py --remote --json

Require exit code 0 and top-level "ok": true. Review the reported host count/groups against the approved inventory. The GPU PCI count is informational; all other failed checks must be understood and resolved before deployment.

Slurm path

ansible-playbook -l slurm-cluster playbooks/slurm-cluster.yml

Rerun the same command after a transient mirror/package error. Validate on a Slurm controller, login, or compute node:

python3 scripts/validation/validate_slurm.py --json

Require exit code 0, "ok": true, "nodes_unavailable": 0, "gpus_configured" greater than zero, and "gpu_job_ok": true.

Kubernetes path

Run only after all Kubernetes package, file, chart, and image references have been resolved to the isolated environment:

ansible-playbook -l k8s_cluster playbooks/k8s-cluster.yml

Validate from a machine whose kubectl context reaches the cluster, overriding the public default CUDA image with the imported registry image:

python3 scripts/validation/validate_k8s.py --json --cuda-smoke \
  --cuda-image registry-host:5000/nvidia/cuda:12.4.1-base-ubuntu22.04

Require exit code 0, "ok": true, nodes_ready == nodes_total, "gpus_allocatable" greater than zero, "cuda_smoke_ok": true, and "cuda_smoke_gpus" greater than zero. A play recap with failed=0 is not the success gate.

Cleanup

• On Kubernetes smoke success, the validator deletes the temporary deepops-validate namespace automatically. On failure it deliberately keeps the namespace for diagnosis; delete it only after collecting evidence:

  kubectl delete namespace deepops-validate --ignore-not-found

• Unmount/eject approved transfer media and remove temporary extracted copies only after checksums, repository reachability, image pulls, and the required validation record have been captured:

  sudo umount /mnt/deepops-packages
  sudo umount /mnt/deepops-images
  sudo umount /mnt/deepops-charts
  sudo umount /mnt/deepops-kubespray
  sudo umount /mnt/deepops-keys

  Do not remove the internal mirrors or registry: deployed nodes continue to need them for repairs and rescheduling.

• Remove secrets and site inventory from transfer staging. Retain the manifest, checksums, exact DeepOps commit, mirror snapshot/version, play recap, and validator JSON according to site policy.

• Delete /tmp/kubespray-effective-vars.json after artifact collection. It is a resolved inventory export and may contain site variables that do not belong in the transfer set or retained validation evidence.

Source-derived failure branches

These branches follow from checked-in tasks, defaults, and validators; they are not claims that this complete procedure was live-tested in an air gap.

• build_offline_cache.sh fails because playbooks/airgap/build-offline-cache.yml is missing: this automation was removed as broken/unsupported. Use the explicit mirror workflow above.
• Doctor reports kubespray_submodule_initialized: false, or Kubernetes fails on kubespray_defaults imports: transfer an archive made after git submodule update --init --recursive; do not fetch from the isolated side.
• Generated Kubernetes lists contain a default version instead of the configured override: confirm that the representative host resolves the override, regenerate its effective-variable JSON, and pass it with -e @.... The generator's localhost play does not inherit k8s_cluster variables from -i alone.
• An image push reports an HTTPS response error against the fallback HTTP registry: perform the import on the registry host through 127.0.0.1:5000. For a remote registry, use trusted TLS or stop until the importer's runtime is deliberately configured and restarted for that exact endpoint; the target-node docker_insecure_registries value does not change the importer.
• Package task tries a public URL or reports a missing package: the mirror set or variable override is incomplete for the enabled profile. Add the exact repository/file to the connected-side manifest, transfer a new signed snapshot, and rerun the same idempotent playbook.
• Slurm play reaches GitHub, SchedMD, Open MPI, or NVIDIA download hosts: mirror and override the applicable source URL, or intentionally disable the optional NHC, Pyxis, HPC SDK, or monitoring component.
• Bare nvidia-smi over SSH reports No devices were found on a Slurm node: DeepOps may hide GPUs from ordinary SSH sessions. Trust the validator's srun job; do not diagnose the driver from the bare SSH result.
• Slurm validator reports gpu_job_ok: false: inspect its failures list. If it reports driver errors, follow skills/diagnose-driver-install/; if the node is down/drained, inspect its Slurm reason before resuming it.
• Kubernetes play tries raw.githubusercontent.com, charts.helm.sh/stable, a public chart repo, or a public image registry: stop. The pinned Helm/chart/image dependency is not yet served internally; complete that mirror or gate the required DeepOps change before rerunning.
• Kubernetes CUDA smoke is Pending or shows an image pull error: confirm the --cuda-image path exists in the internal registry and is reachable by every GPU node. Inspect the retained deepops-validate namespace before cleanup.
• Kubernetes reports gpus_allocatable: 0: wait for the mirrored GPU stack to become ready, then diagnose GPU Operator/device-plugin or driver failures. Do not use --allow-no-gpus as a deployment success gate for a GPU cluster.
• A rerun uses facts from a reprovisioned/reused hostname: add --flush-cache to the same playbook command.