docker-images

Imported from gperdrizet/docker-images (AGENTS.md). Install upstream with npx skills add gperdrizet/docker-images. Copyright stays with the author.

AGENTS.md — Project orientation for AI assistants

What this project is

Twelve containerized ML/AI development environments published to DockerHub as gperdrizet/<family>-<platform>. Four image families × three platforms:

IMPORTANT — deeplearning-nvidia is different from all other nvidia images:

The deeplearning family uses Python 3.10 (forced by the 24.08 NGC base) while all other families use Python 3.12. This is intentional — see WIP section below.

Repository layout

<family>/<platform>/
    Dockerfile              # runtime image
    requirements.txt        # pip dependencies
    motd.sh                 # login banner
    bash.bashrc             # (cpu/mac only) replaces /etc/bash.bashrc
    README.md               # pushed to DockerHub
    Dockerfile.build-*      # (nvidia only) custom wheel builder images
deeplearning/nvidia/
    Dockerfile.build-pytorch
    Dockerfile.build-tensorflow
datascience/nvidia/
    Dockerfile.build-cupy
tests/
    test-<family>-<platform>.sh
wheels/
    *.whl                   # extracted custom wheels (not committed to git;
                            #   uploaded to GitHub Releases and fetched by URL)
Makefile                    # all build / test / push / wheel / release targets

Build workflow

Standard images

make build-<family>-<platform>   # e.g. make build-deeplearning-nvidia
make test-<family>-<platform>
make push-<family>-<platform>

Custom wheel images

Custom wheels are built from source and uploaded to GitHub Releases so the Dockerfiles can fetch them by URL at build time:

Wheel builds are expensive (PyTorch: 3–4 h, CuPy: ~1 h). Only rebuild when the CUDA version, Python version, or GPU arch list changes.

The cp312 wheels target CUDA 12.8 / sm_60–sm_100 (Pascal through Blackwell). The cp310 wheel targets CUDA 12.4 / sm_60–sm_90 (Pascal through Hopper).

Makefile release targets

make release-pytorch-wheel     # upload cp312 pytorch wheel to GitHub Releases
make release-tensorflow-wheel  # upload custom TF wheel (parked, see WIP section)
make release-cupy-wheel        # upload cupy wheel to GitHub Releases

Version bumping

VERSION defaults to the most recent git tag (e.g. v7.0.0 → 7.0.0). Override: make build-all VERSION=7.0.0.

Key design decisions

• deeplearning-nvidia uses NGC base, not bare CUDA: The NVIDIA TF NGC 24.08 image provides pre-compiled TF kernels for Pascal GPUs (CC 6.x). The bare CUDA images do not. This is why deeplearning is an exception to the "all nvidia images share the same base" rule.
• All other nvidia images share the bare CUDA base: nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04. This cannot be swapped for python:3.12-slim because CUDA is a system-level runtime.
• Python path consistency within each family:
  • deeplearning: Python 3.10 at /usr/bin/python3 (NGC), /usr/local/bin/python (cpu/mac)
  • Other families: Python 3.12 with /usr/local/bin/python symlink everywhere
• EXTERNALLY-MANAGED removed: All non-deeplearning nvidia images delete this file. deeplearning-nvidia uses the NGC base which manages this differently.
• Pip upgrade uses --ignore-installed: The apt-installed pip has no RECORD file; --ignore-installed skips the uninstall step that would otherwise fail.
• Custom wheels use --no-deps --force-reinstall: Prevents pip from swapping out the custom PyTorch/CuPy wheels during dependency resolution.
• No CuPy in deeplearning images: Only datascience and kaggle images include CuPy. No cp310 CuPy wheel exists; this will be addressed when deeplearning migrates to 3.12.

Consistency rules (enforced across all images)

• All images create a vscode user (uid/gid 1000) with passwordless sudo.
• HF_HOME is set in .bashrc for all images (huggingface cache location).
• procps must be in every apt block (provides free, used by motd.sh).
• libopenmpi3t64, libcusparselt0, libgomp1 must be in every non-deeplearning nvidia apt block (runtime dependencies of the PyTorch cp312 wheel).
• DEBIAN_FRONTEND=noninteractive and LD_LIBRARY_PATH must be set in every non-deeplearning nvidia image.

Deeplearning package versions (Python 3.10, all three platforms)

Work in progress — deeplearning Python 3.12 migration

deeplearning-nvidia uses the NVIDIA TF NGC 24.08 image (nvcr.io/nvidia/tensorflow:24.08-tf2-py3) as its base. This provides pre-compiled TF 2.17.x kernels for Pascal GPUs (CC 6.0/6.1) and requires Python 3.10. The entire deeplearning family is kept at Python 3.10 for consistency.

Why not a newer NGC image?

• 25.01 and 25.02 NGC images use TF 2.17 but with CC 7.5+ runtime check — Pascal GPU is discovered but JIT compilation crashes the Jupyter kernel on first use.
• The plan was to migrate to a bare CUDA base with a custom TF wheel built from source with explicit Pascal arch targets, but this hit a blocker: see below.

Blocking issue: LLVM CallbackVH hidden symbols

The custom TF 2.20.0 wheel fails to import with:

ImportError: lib_pywrap_tensorflow_common.so: undefined symbol: _ZN4llvm10CallbackVH6anchorEv

Root cause: TF's linker version script (local: * default) hides LLVM symbols needed by lib_pywrap_tensorflow_common.so. The symbols ARE defined in libtensorflow_cc.so.2 but with local (t) visibility. --dynamic-list, --copt=-fvisibility=default, using the tensorflow/build Docker container, and runtime stubs were all attempted and failed.

Definitive fix (not yet implemented): Modify TF's source-level linker version script to explicitly export the CallbackVH symbols, then rebuild the wheel (~6 h). The relevant file is somewhere in tensorflow/core/ BUILD rules. Until then, deeplearning remains on NGC 24.08.

See also: /memories/repo/tensorflow-llvm-anchor-stub.md

Python 3.10 EOL note: Python 3.10 reaches EOL October 2026. This migration needs to happen before then. The path is: fix the TF wheel build → migrate deeplearning-nvidia to bare CUDA base + custom TF wheel → switch deeplearning/cpu and mac back to Python 3.12.

Testing

make test-deeplearning-nvidia   # torch import, tf import, tf GPU kernel on Pascal
make test-deeplearning-cpu
make test-nvidia                # all four nvidia images
make test-all

The tensorflow GPU kernel test runs an actual tf.matmul on the device — this is the Pascal compatibility gate. For deeplearning-nvidia, the first run after a container rebuild may take 1–5 min for PTX JIT on smaller ops, but should not crash.

What this project is

Twelve containerized ML/AI development environments published to DockerHub as gperdrizet/<family>-<platform>. Four image families × three platforms:

Repository layout

<family>/<platform>/
    Dockerfile              # runtime image
    requirements.txt        # pip dependencies
    motd.sh                 # login banner
    bash.bashrc             # (cpu/mac only) replaces /etc/bash.bashrc
    README.md               # pushed to DockerHub
    Dockerfile.build-*      # (nvidia only) custom wheel builder images
deeplearning/nvidia/
    Dockerfile.build-pytorch
    Dockerfile.build-tensorflow
datascience/nvidia/
    Dockerfile.build-cupy
tests/
    test-<family>-<platform>.sh
wheels/
    *.whl                   # extracted custom wheels (not committed to git;
                            #   uploaded to GitHub Releases and fetched by URL)
Makefile                    # all build / test / push / wheel / release targets

Build workflow

Standard images

make build-<family>-<platform>   # e.g. make build-deeplearning-nvidia
make test-<family>-<platform>
make push-<family>-<platform>

Custom wheel images (nvidia only)

Three wheels are built from source and uploaded to GitHub Releases so the Dockerfiles can fetch them by URL at build time:

Wheel builds are expensive (PyTorch: 3–4 h, TensorFlow: ~6 h, CuPy: ~1 h). Only rebuild when the CUDA version, Python version, or GPU arch list changes.

All three wheels target CUDA 12.8 / Python 3.12 / sm_60–sm_100 (Pascal through Blackwell) because NVIDIA's prebuilt wheels drop Pascal (GTX 10xx / P100) support.

Version bumping

VERSION defaults to the most recent git tag (e.g. v4.1.0 → 4.1.0). Override: make build-all VERSION=4.1.0.

Key design decisions

• All nvidia images share the same base: nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04. This cannot be swapped for python:3.12-slim because CUDA is a system-level runtime that must be the foundation; it cannot be layered on top.
• Python path consistency: All images expose /usr/local/bin/python. On nvidia images this is a symlink to /usr/bin/python3.12; on cpu/mac images Python lives there natively.
• EXTERNALLY-MANAGED removed: All nvidia images delete /usr/lib/python3.12/EXTERNALLY-MANAGED to allow bare pip install without --break-system-packages.
• Pip upgrade uses --ignore-installed: The apt-installed pip has no RECORD file; --ignore-installed skips the uninstall step that would otherwise fail.
• Custom wheels use --no-deps --force-reinstall: Prevents pip from swapping out the custom PyTorch/CuPy wheels during dependency resolution.
• TensorFlow is installed without --no-deps: TF has many Python-level dependencies (keras, protobuf, grpcio) that must be resolved normally.

Consistency rules (enforced across all images)

• All images create a vscode user (uid/gid 1000) with passwordless sudo.
• HF_HOME is set in .bashrc for all images (huggingface cache location).
• procps must be in every apt block (provides free, used by motd.sh).
• libopenmpi3t64, libcusparselt0, libgomp1 must be in every nvidia apt block (runtime dependencies of the PyTorch wheel).
• DEBIAN_FRONTEND=noninteractive and LD_LIBRARY_PATH must be set in every nvidia image.
• All three validation RUN python3 -c "import X" checks must pass at build time.

Work in progress — deeplearning-nvidia base image switchover

deeplearning-nvidia was previously based on the official NVIDIA TensorFlow NGC image (nvcr.io/nvidia/tensorflow:25.02-tf2-py3). It is being migrated to the shared bare CUDA runtime base (nvidia/cuda:12.8.1-cudnn-runtime-ubuntu24.04) used by all other nvidia images.

Status as of 2026-08-25: Migration is complete at the Dockerfile level. The image builds and all three framework imports validate successfully. The new TF wheel (2.20.0, built from source) has been uploaded to GitHub Releases and is wired into the Dockerfile.

Known issue: LLVM CallbackVH::anchor stub

Symptom: ImportError: lib_pywrap_tensorflow_common.so: undefined symbol: _ZN4llvm10CallbackVH6anchorEv

Cause: The TF 2.20 bazel build uses --config=opt which enables --gc-sections. llvm::CallbackVH::anchor() is an empty vtable-anchor function that gets stripped as "unreferenced" by the linker, but lib_pywrap_tensorflow_common.so was compiled against system clang 18 headers that reference it.

Current workaround (in deeplearning/nvidia/Dockerfile): A one-function stub .so is compiled at image build time and injected via LD_PRELOAD.

Permanent fix (requires ~6 h TF wheel rebuild): Add to the bazel command in deeplearning/nvidia/Dockerfile.build-tensorflow:

--linkopt="-Wl,--undefined=_ZN4llvm10CallbackVH6anchorEv"

Then remove the stub block and ENV LD_PRELOAD from the runtime Dockerfile, rebuild the wheel, upload to GitHub Releases, and rebuild the container image.

See also: /memories/repo/tensorflow-llvm-anchor-stub.md

Testing

make test-deeplearning-nvidia   # torch, torch CUDA, tensorflow, tensorflow GPU kernel, ...
make test-deeplearning-cpu
make test-nvidia                # all four nvidia images
make test-all

The tensorflow GPU kernel test runs an actual tf.matmul on the device — this is the Pascal compatibility gate for the custom TF wheel.