Imported from asapacsin/llm_quantization (AGENTS.md). Install upstream with npx skills add asapacsin/llm_quantization. Copyright stays with the author.

Agent Memory — Llama 3.1-8B Quantization Benchmark

Read this file first in every new session, then read memory/state.md for where work currently stands.

Project goal

Benchmark Llama 3.1-8B-Instruct under three configurations — FP16 baseline, RTN (calibration-free W4A16), and AWQ (W4A16, calibrated on 512 UltraChat samples) — on accuracy (HellaSwag primary, ARC-Challenge secondary, GSM8K optional) and efficiency (first-token latency, ms/token, tokens/s, peak VRAM, disk size). Core research question: does calibration (AWQ) beat calibration-free RTN at the same bit-width, and at what cost?

GPTQ is out of scope (cost cut); see memory/decisions.md.

Non-negotiable constraints

1. Local machine (Windows, RTX 4050 6GB) is Phase 1 only: write/edit code, configs, and docs. NEVER download models or datasets here, NEVER launch quantization/eval jobs here. 8B FP16 does not fit in 6GB anyway.
2. Phase 2 runs on a RunPod/Vast.ai Linux pod (24GB+ VRAM, 100GB+ disk).
3. Downloads use ModelScope (model.ms_id, calibration.dataset in experiment.yaml). No HF_TOKEN. Downstream scripts load only local_dir.
4. Heavy scripts are guarded (scripts/common.py): download needs --allow-download; GPU scripts exit unless CUDA is available. Keep those guards intact. All scripts support --dry-run (config parse + plan print, no network/GPU) — use it to test locally.
5. All experimental knobs live in configs/experiment.yaml. Never hardcode model ids, sample counts, decoding params, or batch sizes in scripts.

Tooling decisions (rationale in memory/decisions.md)

• Quantization: llm-compressor for both schemes (RTN = QuantizationModifier with no calib data; AWQ = AWQModifier with the calib set). Same library, same W4A16 scheme, same export format -> the only variable is calibration / algorithm.
• Accuracy: lm-eval harness with the vLLM backend, --log_samples on (needed for failure analysis).
• Inference/efficiency: vLLM offline API for every method, greedy decoding.

File map

• configs/experiment.yaml — single source of truth.
• scripts/common.py — config loader, path resolution, guards. Import this in every script.
• scripts/00..09 — numbered in execution order (04 GPTQ removed); README has the full table.
• run_all.sh — Phase 2 orchestrator.
• results/<method>/accuracy/, results/<method>/efficiency.json, results/summary/ — outputs. Method names: fp16, rtn, awq.
• memory/state.md — living checklist. Update it after every work session and after each Phase 2 milestone.
• memory/decisions.md — append-only decision log. Add an entry whenever a tooling/hyperparameter choice is made or changed.

How to resume work

1. Read memory/state.md — find the first unchecked item.
2. If local: only code/doc tasks are allowed (see constraints).
3. If on the cloud pod: bash scripts/00_setup_cloud.sh if fresh, then follow the checklist / run_all.sh. Verify nvidia-smi shows 24GB+ first.
4. After finishing anything, tick it off in memory/state.md and note surprises or deviations there.