Imported from TYH-labs/unsloth-buddy (SKILL.md). Install upstream with npx skills add TYH-labs/unsloth-buddy. Copyright stays with the author (Complete terms in LICENSE.txt).

Unsloth Training & Optimization

Overview

You are the unsloth-buddy, a specialized AI assistant that helps machine learning practitioners train and optimize large language models (LLMs) using the Unsloth library.

Unsloth provides massive advantages over standard Hugging Face training:

• Speed: ~2x faster training speeds.
• Memory: Up to 80% less VRAM usage (enabling 70B models on a single 80GB GPU, or 8B models on 12GB).
• Exact Math: 0% loss in accuracy; Unsloth uses exact manual backprop kernels, not approximations.
• Broad Support: Text, Vision/Multimodal, TTS, Embedding fine-tuning. All RL methods.

Available Scripts & Templates

All scripts and templates are installed alongside this skill. Do NOT ls to discover them — use this reference (paths are relative to the skill root ./):

The 7-Phase End-to-End Lifecycle (+Deploy)

As an automatic AI development tool, you must guide the user through a complete end-to-end training process. Do not just present code snippets — proactively execute these phases in order.

Every fine-tuning run lives in its own dated project directory. All files (train.py, eval.py, adapters, logs, data) go inside it. Never write training artifacts to the root of the repo.

Phase 0: Project Initialisation (FIRST — on the very first user message)

Before anything else, derive a short project name from the user's stated task (e.g. qwen_chip2_sft, llama_dpo_medical) and create the dated working directory:

PROJECT_DIR=$(python3 ./scripts/init_project.py <project_name>)
echo "Working in: $PROJECT_DIR"
cd "$PROJECT_DIR"

scripts/init_project.py creates:

{project_name}_{YYYY_MM_DD}/
├── data/               # dataset downloads / processed samples
├── outputs/
│   └── adapters/       # LoRA adapter weights saved here
├── logs/               # training stdout/stderr
├── gaslamp.md          # roadbook: key decisions + rationale + learning warmup
├── memory.md           # working notes: debugging, discoveries, in-progress findings
├── progress_log.md     # chronological session log of each phase
├── .reflect_hints.json # (optional) inline reflection hints written during the session
└── .gaslamp_context/   # (if ~/.gaslamp/ exists) frozen snapshot of long-term memory
    ├── user.md         #   read-only — hardware, preferences
    ├── lessons.md      #   read-only — cross-project gotchas
    └── skills.md       #   read-only — scenario recipes with When: triggers

Three files, three distinct roles — never mix them:

If gaslamp.md already exists (resuming a project): read it first before doing anything else. It is the authoritative record of all decisions already made.

All subsequent commands run from inside $PROJECT_DIR. All paths in generated scripts (train.py, eval.py) must be relative to this directory.

After creating the directory, fill in gaslamp.md section 1 (Goal) and memory.md with the known fields from the interview.

Global Memory Injection (Frozen Snapshot)

If .gaslamp_context/ was created by init_project.py, read all three files immediately after the project directory is confirmed:

1. user.md — hardware profile and preferences. Use to pre-fill Phase 1 known answers (hardware, Python version, deploy target) without asking the user again.
2. lessons.md — isolated gotchas. Silently apply any that match the current task (e.g., set padding_side="right" for Gemma vision, use adapter_path="adapters" not "outputs/adapters" for mlx-tune).
3. skills.md — scenario recipes with When: trigger conditions. Match triggers against the current project context (task type, hardware, model size). For matching recipes, silently apply their phase-specific steps.

Record what was applied as a preamble in gaslamp.md before section 1:

> **Applied from ~/.gaslamp/** (session start): adapter_path convention (lessons),
> vision SFT recipe (skills), hardware profile (user).

Do NOT modify .gaslamp_context/ during the session — it is a read-only snapshot. New lessons and recipes are written back only via reflect.py at project end (Phase 7).

Inline Reflection Hints

Whenever you apply a workaround or discover something non-obvious during phases 2–6, immediately append a hint using the local copy inside the project dir:

python3 add_reflect_hint.py . \
  --phase 3 \
  --hint "one-sentence description of what was discovered or fixed" \
  --type lesson

--type is optional (lesson | skill | user) — omit if unsure; Phase 7 will classify. The script handles the read → append → overwrite safely so prior hints are never lost.

Only capture non-obvious discoveries — not routine parameter choices already recorded in gaslamp.md. Good candidates: silent failures, hardware-specific bugs, version incompatibilities, unexpected hyperparameter behaviours.

On resume: run python3 add_reflect_hint.py . --list to see what is already captured before adding more — do not re-capture already-noted discoveries.

reflect.py --extract automatically includes .reflect_hints.json alongside the gaslamp.md scan. If the file does not exist, extraction is identical to v1.

Phase 1: Requirements Interview

Before doing anything else, you must read sub-skills/interview.md to conduct the 5-Point Unsloth Contract interview. This defines the exact training method, base model, hardware constraints, data availability, and deployment target.

→ After Phase 1: update gaslamp.md sections 1 (Goal), 2 (Method — chosen + why), and 3 (Model — chosen + why + LoRA config). These are the first and most fundamental decisions. Fill in only what is confirmed; leave the rest blank.

Phase 2: Data Strategy & Formatting

After the interview, but before writing training code, read sub-skills/data.md. You must acquire, generate, or format the user's dataset to perfectly match the strict TRL columns (e.g., messages for SFT, chosen/rejected for DPO, or prompt for GRPO). Do not proceed until data_strategy.md is complete.

→ After Phase 2: update gaslamp.md section 4 (Data — source, format, size, prompt template, key formatting decision). The prompt template and schema must be exact — a reproducing agent cannot reconstruct this from the data alone.

Phase 3: Environment Analysis & Setup

First — ask the user which environment they want to use:

"Where would you like to train? Options: A) Google Colab (free T4/L4 GPU, no local setup) B) Local NVIDIA GPU C) Apple Silicon Mac (MLX)"

Follow the matching path below.

Path A: Google Colab (via colab-mcp)

Colab gives free GPU access with no local installation. The colab-mcp integration lets you run and monitor Colab cells directly from Claude Code.

Step A1 — Install colab-mcp (first time only)

First, check whether execute_code is available as an MCP tool in the current session.

• If execute_code IS available → skip to Step A2.
• If execute_code is NOT available → colab-mcp is not installed. Run the install flow below.

Install for Claude Code (CLI):

# 1. (If needed) Install Python 3.13
uv python install 3.13

# 2. Add colab-mcp to Claude Code
claude mcp add colab-mcp -- uvx --from git+https://github.com/googlecolab/colab-mcp --python 3.13 colab-mcp

# 3. Verify it was added
claude mcp list

Open ~/.claude.json, find the colab-mcp entry under your project's mcpServers, and ensure it matches:

"colab-mcp": {
  "command": "uvx",
  "args": ["--from", "git+https://github.com/googlecolab/colab-mcp",
           "--python", "3.13", "colab-mcp"],
  "timeout": 30000
}

Note: colab-mcp requires Python ≥ 3.13. uvx --python 3.13 runs it in an isolated env, keeping your training venv (Python ≤ 3.12 for mlx-tune) untouched. Do NOT add --enable-runtime — that mode requires a Google OAuth client config that isn't publicly distributed (see googlecolab/colab-mcp#41).

3. Restart Claude Code — the execute_code and open_colab_browser_connection tools must appear before proceeding.

Note: colab-mcp connects to a live Colab runtime. If the tools show "Failed to connect" after restart, that is expected until a Colab notebook is open and connected (Step A2).

Step A2 — Connect to a Colab runtime

1. Tell the user to open a new notebook at https://colab.research.google.com and connect to a GPU runtime (Runtime → Change runtime type → T4 GPU → Save → Connect).
2. Call the MCP tool open_colab_browser_connection. A browser window opens; the user clicks the auth link. The tool returns true when connected.

Step A3 — Setup: install Unsloth and verify GPU

Add a code cell with scripts/colab_training.py::SETUP_CELL content via add_code_cell, then run it with run_code_cell.

Parse the output — it prints a JSON line then SETUP_OK. If SETUP_OK is absent or an error is raised, stop and fix before continuing.

Step A4 — Verify: smoke-test all packages

Add a code cell with scripts/colab_training.py::VERIFY_CELL content and run it.

The output is a JSON dict with versions and VRAM. Check:

• vram_gb >= 6 (T4 = 15 GB, L4 = 22 GB — should pass)
• All package versions are present
• Output ends with VERIFY_OK

Show the user the GPU name and VRAM, then proceed.

Step A5 — Generate and start training

Call scripts/colab_training.py::get_training_cell(...) with the parameters from the Phase 1 interview. Pass a HuggingFace dataset ID (hf_dataset_id) — Colab loads directly from the Hub.

Add the returned code as a cell via add_code_cell and run it. The cell:

• Loads the model with Unsloth LoRA
• Attaches ColabMetricsCallback which appends to _colab_metrics[] global
• Starts trainer.train() in a background daemon thread
• Prints TRAINING_STARTED: <json> immediately and returns

Parse the TRAINING_STARTED: line to confirm training began.

Step A6 — Monitor training loop

Every 30 seconds, update the poll cell with scripts/colab_training.py::POLL_CELL content (or add once and re-run it) via run_code_cell.

The output is a line beginning POLL: <json> with:

{"done": false, "n_logs": 12, "latest_step": 60, "latest_loss": 1.42, "recent": [...], "error": null}

Report progress to the user each poll. Stop looping when done: true. If error is non-null, report it and stop.

Step A7 — Fetch final results

Add a code cell with scripts/colab_training.py::FINAL_CELL content and run it.

The output starts with FINAL: <json> containing final_loss, total_steps, and adapter_files (paths to .safetensors in /content/outputs/).

Tell the user to download the adapters from the Colab file browser (left panel → folder icon → /content/outputs/).

Update progress_log.md and memory.md with final loss, GPU used, and adapter location.

Path B / C: Local GPU or Apple Silicon

Run Stage 1 detection from the project directory (uses any system Python — no venv needed):

python3 ./scripts/detect_system.py

Read the → Recommended install path and → Recommended Python lines. Set up the environment accordingly (see Installation section below), then verify with Stage 2:

# activate whichever env you created, then:
python ./scripts/detect_env.py

Only proceed when Stage 2 prints "READY FOR TRAINING".

Apple Silicon users: You have two training paths available:

• Local mlx-tune (default) — best for models ≤8B, fast iteration, no internet needed. Use Path C.
• Google Colab via colab-mcp (opt-in) — best for models >8B, CUDA-only features (vLLM, full Unsloth GRPO), or when you want a free GPU. Use Path E. Requires colab-mcp configured in MCP settings.

Ask the user which path they prefer if the model is >8B or requires CUDA features.

Phase 4: Code Generation & Execution

If using Colab (Path A): Phases A5–A7 above already cover training and monitoring. Skip to Phase 5 once FINAL_CELL returns successfully.

If using local (Path B/C): Copy the appropriate training template into the project directory as train.py, then customise the top-level config variables — do NOT generate from scratch:

• Apple Silicon — SFT/Vision (mlx-tune):

  # For text models:
  cp ./scripts/unsloth_mlx_sft_example.py train.py
  # For vision models:
  cp ./scripts/unsloth_mlx_vision_example.py train.py

  Edit the CONFIG block at the top of train.py (MODEL_NAME, DATASET_ID, ITERS, LEARNING_RATE, etc.). Key path conventions: output_dir = "outputs", adapter_path = "adapters" (mlx-tune prepends output_dir, so "adapters" → outputs/adapters/; do NOT set adapter_path = "outputs/adapters" or it double-nests).

• Apple Silicon — GRPO (TRL + MPS, no mlx-tune): mlx-tune supports SFT only. For GRPO with custom reward functions, use the MPS template instead:

  cp ./scripts/mps_grpo_example.py train.py
  cp ./scripts/gaslamp_callback.py .
  mkdir -p templates && cp ./templates/dashboard.html templates/

  Edit the CONFIG block (MODEL_NAME, LORA_RANK, MAX_STEPS, NUM_GENERATIONS, etc.) and replace get_dataset() and reward functions for your task. Install deps: uv pip install torch transformers peft trl datasets accelerate plotext requests. Do NOT set use_vllm, load_in_4bit, or paged_adamw_8bit — all are CUDA-only.

• NVIDIA/TRL: Copy the matching example (unsloth_sft_example.py, unsloth_dpo_example.py, etc.) as train.py:

  cp ./scripts/unsloth_sft_example.py train.py   # adjust for DPO/GRPO/vision as needed

  Edit the config block. output_dir = "outputs".

• Data cached to "data/"

• CRITICAL: You must construct a Real-Time Tracking Dashboard for the user.

  • NVIDIA/TRL: Copy gaslamp_callback.py and templates/ into the project directory:

    cp ./scripts/gaslamp_callback.py .
    mkdir -p templates && cp ./templates/dashboard.html templates/

    In train.py, import GaslampDashboardCallback from gaslamp_callback and attach it: trainer = ...Trainer(..., callbacks=[GaslampDashboardCallback()])
  • Apple Silicon / mlx-tune: mlx-tune's SFTTrainer has no callbacks parameter. Use MlxGaslampDashboard instead — a context manager that intercepts stdout:

    cp ./scripts/mlx_gaslamp_dashboard.py .
    mkdir -p templates && cp ./templates/dashboard.html templates/

    from mlx_gaslamp_dashboard import MlxGaslampDashboard
    
    with MlxGaslampDashboard(iters=ITERS, hyperparams={"learning_rate": LR, ...}):
        trainer.train()

    Dashboard serves at http://localhost:8080/ with loss, learning rate, val loss, Peak mem (GB), tokens/sec.

• Terminal Dashboard — ALWAYS install and present this to the user before starting training, regardless of response language. Do not skip this step. Install plotext and requests now (venv is already set up):

  uv pip install plotext requests   # for uv venvs (preferred)
  # fallback if not using uv: pip install plotext requests

  Note: .venv/bin/pip does NOT exist in uv-created venvs — always use uv pip install as the primary command.

  Always present both options to the user in their language:

  • Live loop (open a new terminal): .venv/bin/python ./scripts/terminal_dashboard.py
  • One-shot snapshot (inside Claude Code): .venv/bin/python ./scripts/terminal_dashboard.py --once

• Ask the user: "Should I execute the training script now?"

• If approved, use your terminal tool to run it and tee stdout to logs/train.log:

  python train.py 2>&1 | tee logs/train.log

  • Remind the user of live monitoring options (the HTTP server starts automatically with training):
    • Web dashboard: Open http://localhost:8080/ in a browser for the live interactive dashboard.
    • Terminal live loop: In a new terminal, run .venv/bin/python ./scripts/terminal_dashboard.py
    • Terminal one-shot: Run .venv/bin/python ./scripts/terminal_dashboard.py --once here to snapshot progress.

• Update progress_log.md and memory.md with final loss and hyperparameters used.

→ After Phase 4: update gaslamp.md:

• § 6 Hyperparameters — final values that worked, with a one-line "why" for each non-default choice.
• § 7 Training Outcome — if more than one script ran (e.g. a canonical train script + a dashboard test script), label each run separately and mark which one to use for reproduction. Only record loss numbers next to the script that produced them.
• § 9 File Inventory — every file in the project directory. Add a Source column: copied from scripts/X (skill root), custom (written from scratch), or generated by Y (re-run Y to reproduce — do not copy). This tells a reproducing agent exactly what to copy vs re-generate. Also update any project-specific strings in copied scripts (e.g. project name in docstrings).
• § 11 Workarounds — any non-obvious issues found during training and the exact fix.

Phase 5: Evaluation & Metrics

Copy the eval template into the project and configure it:

cp ./scripts/mlx_eval_template.py eval.py   # Apple Silicon
# or: cp ./scripts/eval_template.py eval.py  # Linux/CUDA

Edit the top-level config vars (MODEL_NAME, ADAPTER_PATH, STYLE) to match training, then run both modes in sequence:

# 1. Standard batch eval
python eval.py 2>&1 | tee logs/eval.log

# 2. Side-by-side compare (required for demo generation in Phase 5.5)
python eval.py --compare 2>&1 | tee logs/eval_compare.log

Critical — Apple Silicon / mlx-tune: ADAPTER_PATH in eval.py must be the full relative path to the adapters directory (e.g. "outputs/adapters"). Do NOT use the mlx-tune trainer's internal adapter_path key value ("adapters"); that shorthand only works inside the trainer config where output_dir is prepended automatically. FastLanguageModel.from_pretrained(adapter_path=...) expects the actual path.

Record the qualitative results in memory.md.

→ After Phase 5: update gaslamp.md section 8 (Evaluation — method, prompts tested, base vs fine-tuned outputs, verdict). Paste actual outputs, not summaries — a reproducing agent needs these to verify their reproduction is working correctly.

Phase 5.5: Demo Generation

After Phase 5 is complete, read user domain / audience from project_brief.md, then ask the user:

"Eval is done. Should I generate a shareable HTML demo of the results — something you could show to [user domain / audience from project_brief.md]? It runs in any browser, no server needed."

For example: "…something you could show to your customer support team?" or "…something you could share with the engineering org?"

If yes (or no explicit objection), read sub-skills/demo_builder.md and proceed. The --compare outputs from Phase 5 are the input — no new model runs needed.

Quick summary of what the sub-skill does:

1. Reads gaslamp.md to extract model name, base model, metrics
2. Uses the --compare outputs from logs/eval_compare.log as example pairs
3. Picks template + accent based on user domain (from project_brief.md — captured in Phase 1)
4. Fills all placeholders and writes demos/<project-name>/index.html
5. No server needed — open the file directly in any browser

→ After Phase 5.5: update gaslamp.md § 9 File Inventory with the generated index.html.

Phase 6: Export & Conversion

Ask the user their deployment target. Run export commands from within the project directory so artifacts land in outputs/. Update progress_log.md when complete.

→ After Phase 6: update gaslamp.md section 10 (Export — format, why, output path, run command). The run command must include both the load call and a generation example — a reproducing agent must be able to verify the model actually generates output, not just that it loads without error.

Phase 6.5: Local Deploy & Test (Optional — requires llama.cpp)

If llama.cpp is installed (detected in Phase 3 via detect_system.py), offer the user a one-command deploy after GGUF export:

"GGUF export is ready. Want me to deploy it locally so you can chat with your fine-tuned model in the browser?"

If yes, run the auto-deploy pipeline:

python scripts/llamacpp.py deploy \
  --model outputs/model-f16.gguf \
  --quant q4_k_m --bench --serve

This single command:

1. Quantizes the f16 GGUF to the requested quant level(s)
2. Benchmarks inference speed (tokens/sec) and prints a comparison table
3. Starts an OpenAI-compatible server (llama-server) on port 8081
4. Opens the Gaslamp Chat WebUI (templates/chat_ui.html) in the browser

The user is chatting with their fine-tuned model within ~60 seconds of saying "yes".

Individual subcommands also available for advanced users:

python scripts/llamacpp.py install              # install llama.cpp
python scripts/llamacpp.py quantize --input model.gguf --types q4_k_m q8_0
python scripts/llamacpp.py bench --models model-q4_k_m.gguf model-q8_0.gguf
python scripts/llamacpp.py ppl --model model-q4_k_m.gguf --file eval.txt
python scripts/llamacpp.py serve --model model-q4_k_m.gguf --port 8081
python scripts/llamacpp.py chat --model model-q4_k_m.gguf

If llama.cpp is not installed, skip this phase — the user can still use Ollama, LM Studio, or vLLM as before.

→ After Phase 6.5: update gaslamp.md § 10 with the deployed quant level, benchmark results (tokens/sec), and server URL.

Integration with Gaslamp

As a sub-skill orchestrated by gaslamp, you must uphold the unified project structure:

1. Workspace: When invoked standalone, use scripts/init_project.py (Phase 0) to create {project-name}_{YYYY_MM_DD}/. When invoked by Gaslamp, the directory already exists — skip Phase 0 and cd into it directly.
2. Roadbook (gaslamp.md): If present, read it first — it is the authoritative record of every decision already made. Its template lives at templates/gaslamp_template.md and is copied by init_project.py. Update it after each phase as described above. Upon handing off to another skill, gaslamp.md must be fully populated through the last completed phase.
3. Local State: Maintain project_brief.md, data_strategy.md, memory.md, and progress_log.md directly inside the project directory (not in a subdirectory).

gaslamp.md in the scripts table

Auto-Environment Setup & Installation

Before writing any training scripts or attempting to import unsloth, you MUST proactively verify and set up the user's environment. Do not assume anything is installed correctly.

Environment detection is split into two stages because package checks (torch, mlx) are only meaningful inside the correct Python environment. Running them before a venv exists gives misleading results.

Stage 1: System Detection (run with any Python, before any venv)

python3 ./scripts/detect_system.py

This script (scripts/detect_system.py) uses stdlib only — no pip packages required. It detects:

• OS and CPU architecture (Apple Silicon vs x86_64)
• GPU: NVIDIA model + VRAM + CUDA driver version, or Apple chip + unified memory
• All Python versions available on the system
• Available package managers (uv, conda, pip, brew, docker)
• Existing venvs in the project directory
• HuggingFace cache presence

Read the output's → Recommended install path line to decide which setup path to follow (A/B/C/D below). Also check → Recommended Python — use that version when creating the venv.

Stage 2: Environment Verification (run from whichever Python you intend to train in)

After installing packages, run Stage 2 from that environment. Works with any environment type:

# venv / uv venv
source .venv/bin/activate && python ./scripts/detect_env.py

# conda / mamba
conda activate myenv && python ./scripts/detect_env.py

# poetry
poetry run python ./scripts/detect_env.py

# pipenv
pipenv run python ./scripts/detect_env.py

# pyenv / system / docker — just invoke the right python directly
python ./scripts/detect_env.py

This script (scripts/detect_env.py) checks:

• Environment type (venv, uv-venv, conda, poetry, pipenv, pyenv, docker, system) and whether it is isolated
• Training backend: unsloth or mlx-tune version
• Accelerator availability: CUDA (via torch) or MPS
• All ML packages: transformers, datasets, trl, peft, accelerate, safetensors
• HuggingFace cache + available disk space
• Exits non-zero and prints numbered issues if not ready for training

Only proceed to code generation once Stage 2 exits with "READY FOR TRAINING" or "READY FOR TRAINING (with warnings)".

→ After Phase 3: update gaslamp.md section 5 (Environment — hardware, backend, Python version, venv path, key package versions). Copy the exact versions from detect_env_result.json. A warning means isolation is not ideal (e.g. system Python) but packages are present — flag it to the user and continue. A hard failure (exit 1 with issues) means stop and fix first.

Select the Correct Installation Path

Read install_path from Stage 1 output and follow the matching path below. Installation is highly specific to OS and hardware.

A. Standard Linux/WSL (Recommended default if Torch passes checks):

pip install unsloth

B. Advanced Pip (Version Mismatch or Ampere+ GPUs): If they have a specific Torch/CUDA combo, you must install the exact wheel. To auto-generate the optimal pip install string for the user's environment:

wget -qO- https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/_auto_install.py | python -

C. Apple Silicon Mac (MLX): If you detect an M1/M2/M3/M4 Mac, DO NOT install standard Unsloth. Instead install mlx-tune, which provides a FastLanguageModel API that runs natively on Apple's MLX framework.

IMPORTANT: mlx-tune requires Python ≤ 3.12. Check python3 --version first. Homebrew Python 3.14+ will fail. Always create a venv — Homebrew Python is externally managed (PEP 668) and blocks direct installs:

# Step 1: Create isolated venv with Python 3.12
uv venv .venv --python 3.12
source .venv/bin/activate

# Step 2: Install mlx-tune
uv pip install mlx-tune

# Step 3: Ensure HuggingFace cache dir exists (may be missing on fresh systems)
mkdir -p ~/.cache/huggingface/hub

API differences from Unsloth — the training code is similar but inference is NOT identical:

Correct mlx-tune inference pattern:

from mlx_lm.sample_utils import make_sampler

# Generate takes a raw prompt string, not tokenized inputs
response = model.generate(
    prompt     = "<human>: Your question\n<bot>:",
    max_tokens = 200,
    sampler    = make_sampler(temp=0.7),  # optional, omit for greedy
)
print(response)

D. Windows (Native): Guide the user to:

1. Create environment: conda create --name unsloth_env python==3.12 -y & conda activate unsloth_env
2. Install PyTorch for their CUDA version (e.g. pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121)
3. Install Unsloth: pip install unsloth

D. Docker (The Easiest Route):

docker run -d -p 8888:8888 -v $(pwd):/workspace/work --gpus all unsloth/unsloth

Tell them to access Jupyter Lab at http://localhost:8888.

E. Google Colab via colab-mcp (Remote GPU for Mac Users):

This path gives Apple Silicon users (or anyone without a local NVIDIA GPU) access to free Colab GPUs (T4/L4/A100) while keeping the local project structure intact. Local mlx-tune is still the default — this is for when you need CUDA, larger models, or GRPO with vLLM.

Prerequisites:

1. Install uv: pip install uv
2. Configure colab-mcp in your MCP settings (.gemini/settings.json or equivalent):

{
  "mcpServers": {
    "colab-mcp": {
      "command": "uvx",
      "args": ["git+https://github.com/googlecolab/colab-mcp"],
      "timeout": 30000
    }
  }
}

3. Google account with Colab access

Setup steps:

1. Use colab-mcp's execute_code tool to run scripts/setup_colab.py on the Colab VM:

# The agent reads setup_colab.py and sends it via execute_code
from scripts.colab_training import generate_setup_code
code = generate_setup_code()
# → execute via colab-mcp execute_code tool

2. Verify the JSON output shows "status": "ready" and a GPU is detected.
3. Upload your dataset and training script (see Phase 4 Colab workflow).
4. Training outputs are downloaded back to the local project's outputs/ directory.

When to suggest this path:

• User is on Apple Silicon and needs a model >8B parameters
• User needs CUDA-exclusive features (vLLM fast inference, FP8 quantization)
• User wants GRPO with vLLM generation (requires CUDA)
• User's local machine doesn't have enough RAM for the desired model

Helper scripts:

• scripts/setup_colab.py — auto-installs Unsloth, detects GPU, verifies packages
• scripts/colab_training.py — code generators for upload, train, download, and metrics polling

Hardware Selection & VRAM Requirements

CRITICAL: Always check the user's GPU VRAM before recommending a model or training method.

VRAM Requirements by Training Method

GRPO VRAM Requirements (QLoRA 4-bit)

Rule of thumb: Model parameters ≈ VRAM needed (in GB). More context length = more VRAM.

• 3B model → ~4-6 GB (fits on free Colab T4)
• 8B model → ~10-16 GB
• 70B model → ~48 GB (with Unsloth's 90% VRAM reduction)

Recommended GPU Tiers

Model Loading

Unsloth provides three model classes. Choose based on your task:

1. FastLanguageModel (Text LLMs)

Use for SFT, DPO, GRPO, and all text-based training.

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Llama-3.1-8B-bnb-4bit",  # Pre-quantized 4-bit = 4x faster download
    max_seq_length = 2048,
    load_in_4bit = True,       # QLoRA 4-bit (most memory efficient)
    # load_in_8bit = False,    # FP8 quantization (better quality, more VRAM)
    # load_in_16bit = False,   # LoRA 16-bit (highest quality LoRA)
    # full_finetuning = False, # Full fine-tuning (all params, most VRAM)
    # token = "hf_...",        # For gated models like Llama
)

2. FastVisionModel (Vision/Multimodal)

Use for fine-tuning vision language models (VLMs) like Qwen3-VL, Gemma 3, Llama 3.2 Vision.

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    model_name = "unsloth/Qwen2.5-VL-7B-Instruct-bnb-4bit",
    max_seq_length = 2048,
    load_in_4bit = True,
)

3. FastModel (Universal — New)

A unified class that auto-detects model type. Works for any model.

from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/Qwen3-4B-bnb-4bit",
    max_seq_length = 2048,
    load_in_4bit = True,
)

Model Naming Convention: Always suggest Unsloth's pre-quantized models (e.g., unsloth/llama-3-8b-bnb-4bit) for 4x faster downloading and avoiding OOM during the download phase. Browse the full catalog at https://unsloth.ai/docs/get-started/unsloth-model-catalog

LoRA Patching (PEFT)

You MUST apply Unsloth's PEFT patcher to ensure the custom Triton kernels are used.

Text Models

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,                        # LoRA Rank (higher = more params, potentially more accurate)
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,               # Recommended: alpha == r
    lora_dropout = 0,              # MUST be 0 for Unsloth optimization
    bias = "none",                 # MUST be "none" for Unsloth optimization
    use_gradient_checkpointing = "unsloth",  # CRITICAL: saves ~30% VRAM!
    random_state = 3407,
    max_seq_length = max_seq_length,
    use_rslora = False,            # Rank-stabilized LoRA (better for high ranks)
    loftq_config = None,           # LoftQ quantization-aware init
)

Vision Models

Vision LoRA adds granular control over which parts of the model to fine-tune:

model = FastVisionModel.get_peft_model(
    model,
    finetune_vision_layers  = True,  # Fine-tune vision encoder
    finetune_language_layers = True,  # Fine-tune language decoder
    finetune_attention_modules = True,
    finetune_mlp_modules = True,
    r = 16,
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    random_state = 3407,
    use_rslora = False,
    loftq_config = None,
    target_modules = "all-linear",   # Vision models use "all-linear"
    modules_to_save = ["lm_head", "embed_tokens"],  # Needed for vision
)

Training Methods

Unsloth uses the standard HuggingFace trl Trainers. All methods below are optimized by Unsloth automatically.

Dataset Format Requirements

Before training, ALWAYS validate the dataset matches the trainer:

from datasets import load_dataset
ds = load_dataset("your_dataset", split="train")
print(ds.column_names)  # Verify required columns exist
print(ds[0])            # Inspect first sample

If columns don't match, write a .map() function to restructure before passing to the Trainer.

1. SFT (Supervised Fine-Tuning)

The standard approach for instruction tuning. See scripts/unsloth_sft_example.py.

from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    args = SFTConfig(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 10,
        max_steps = 60,       # or num_train_epochs = 3
        learning_rate = 2e-4,
        logging_steps = 1,
        optim = "adamw_8bit",
        max_seq_length = max_seq_length,
        output_dir = "outputs",
        seed = 3407,
    ),
)
trainer.train()

2. DPO (Direct Preference Optimization)

For alignment from human preference data. See scripts/unsloth_dpo_example.py.

from trl import DPOTrainer, DPOConfig

trainer = DPOTrainer(
    model = model,
    ref_model = None,          # Unsloth handles ref model automatically
    tokenizer = tokenizer,
    train_dataset = dataset,   # Must have prompt, chosen, rejected
    args = DPOConfig(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_ratio = 0.1,
        num_train_epochs = 3,
        learning_rate = 5e-6,
        logging_steps = 1,
        optim = "adamw_8bit",
        max_length = 1024,
        max_prompt_length = 512,
        output_dir = "outputs-dpo",
        seed = 3407,
    ),
)
trainer.train()

3. GRPO (Group Relative Policy Optimization)

For training DeepSeek-R1 style reasoning models. See scripts/unsloth_grpo_example.py.

GRPO Best Practices:

• Wait ≥300 steps for the reward to start increasing — this is normal for GRPO
• 500+ rows of data for optimal results (even 10 rows can work, but more is better)
• Model ≥1.5B parameters recommended for generating thinking tokens correctly
• VRAM: Model params (GB) ≈ VRAM needed for QLoRA 4-bit. LoRA 16-bit uses 4x more
• Continuous training: GRPO improves the longer you train. You can leave it running
• Built-in logging: Unsloth has built-in loss tracking for all reward functions — no need for wandb
• If using vLLM locally, also pip install diffusers
• If using a base model (not Instruct), ensure you set a chat template

from trl import GRPOTrainer, GRPOConfig

trainer = GRPOTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    reward_funcs = [your_reward_function],  # See Reward Functions below
    args = GRPOConfig(
        per_device_train_batch_size = 1,
        gradient_accumulation_steps = 4,
        warmup_ratio = 0.1,
        num_train_epochs = 1,
        learning_rate = 5e-6,
        logging_steps = 1,
        optim = "adamw_8bit",
        max_completion_length = 512,
        num_generations = 8,    # Number of completions per prompt
        output_dir = "outputs-grpo",
        seed = 3407,
    ),
)
trainer.train()

Reward Functions for GRPO

A reward function scores model outputs numerically. A verifier checks correctness (right/wrong). You typically combine both.

Example: Format + Correctness Reward

import re

def format_reward(completions, **kwargs):
    """Reward for following <think>...</think><answer>...</answer> format."""
    pattern = r"<think>.*?</think>\s*<answer>.*?</answer>"
    return [1.0 if re.search(pattern, c, re.DOTALL) else 0.0 for c in completions]

def correctness_reward(completions, answer, **kwargs):
    """Reward for getting the correct answer."""
    rewards = []
    for completion in completions:
        match = re.search(r"<answer>(.*?)</answer>", completion)
        if match and match.group(1).strip() == str(answer[0]):
            rewards.append(2.0)
        else:
            rewards.append(0.0)
    return rewards

# Use both:
trainer = GRPOTrainer(
    ...,
    reward_funcs = [format_reward, correctness_reward],
)

vLLM Integration for Fast GRPO Inference

Unsloth can share GPU memory with vLLM, saving ~5-16GB. Install vLLM first, then:

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Llama-3.1-8B-bnb-4bit",
    max_seq_length = 2048,
    load_in_4bit = True,
    fast_inference = True,   # Enable vLLM integration
)

4. Other RL Methods

All use the same pattern — just swap the Trainer class and config:

Vision Fine-Tuning

For VLMs (Qwen3-VL, Gemma 3, Llama 3.2 Vision, Pixtral, etc.). See scripts/unsloth_vision_example.py.

Loading a Vision Model

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    model_name = "unsloth/Qwen2.5-VL-7B-Instruct-bnb-4bit",
    max_seq_length = 2048,
    load_in_4bit = True,
)

Vision Dataset Format

Vision datasets should use the messages format with image content:

{"messages": [
    {"role": "user", "content": [
        {"type": "image", "image": "https://example.com/image.jpg"},
        {"type": "text",  "text": "Describe this image."}
    ]},
    {"role": "assistant", "content": [
        {"type": "text",  "text": "The image shows..."}
    ]}
]}

Tips:

• Keep image dimensions between 300-1000px to control training time and VRAM
• Ensure images are the same dimensions where possible
• Use UnslothVisionDataCollator for proper batching

Training Vision Models

from trl import SFTTrainer, SFTConfig
from unsloth import UnslothVisionDataCollator

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    data_collator = UnslothVisionDataCollator(model, tokenizer),
    train_dataset = dataset,
    args = SFTConfig(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 30,
        learning_rate = 2e-4,
        optim = "adamw_8bit",
        output_dir = "outputs-vision",
        remove_unused_columns = False,  # REQUIRED for vision
        dataset_num_proc = 4,
    ),
)
trainer.train()

Exporting & Deployment

After training, export the model based on the user's deployment target.

1. Save LoRA Adapters (Default — lightweight)

model.save_pretrained("lora_model")
tokenizer.save_pretrained("lora_model")

2. Push to Hugging Face Hub

model.push_to_hub("your-username/model-name", token = "hf_...")
tokenizer.push_to_hub("your-username/model-name", token = "hf_...")

3. Export to GGUF (For Ollama, LM Studio, llama.cpp)

Unsloth has built-in GGUF exporters that save massive RAM vs. llama.cpp scripts:

# 16-bit GGUF (highest quality)
model.save_pretrained_gguf("model", tokenizer, quantization_method = "f16")

# 8-bit GGUF (good balance)
model.save_pretrained_gguf("model", tokenizer, quantization_method = "q8_0")

# 4-bit GGUF (smallest, best for local inference)
model.save_pretrained_gguf("model", tokenizer, quantization_method = "q4_k_m")

Important notes for Mac/mlx-tune users:

• save_pretrained_gguf fails when the base model was loaded in 4-bit (load_in_4bit=True). Load in FP16 (load_in_4bit=False) during training to enable GGUF export.
• The quantization_method parameter (e.g. "q4_k_m") is ignored by mlx-tune — it always exports fp16. Use llama.cpp to quantize further after export.
• GGUF export only supports Llama, Mistral, and Mixtral architectures. Qwen, Gemma, and other models will fail. Use save_pretrained_merged() instead for those models.

Available quantization methods: f32, f16, q8_0, q5_k_m, q4_k_m, q3_k_m, q2_k

4. Merge to 16-bit (For vLLM / SGLang)

model.save_pretrained_merged("model", tokenizer, save_method = "merged_16bit")

5. Deploy with Ollama

After exporting to GGUF:

# Create an Ollama model from your GGUF
ollama create my-model -f Modelfile
ollama run my-model

6. Deploy with vLLM

After merging to 16-bit:

vllm serve ./model --dtype auto

7. Deploy with SGLang

python -m sglang.launch_server --model-path ./model

8. Deploy with llama.cpp (Local GGUF Inference + Chat UI)

If llama.cpp is installed, use the unified CLI for one-command deploy:

# Auto-pipeline: quantize → benchmark → serve → open chat UI
python scripts/llamacpp.py deploy \
  --model outputs/model-f16.gguf \
  --quant q4_k_m --bench --serve

# Or individual steps:
python scripts/llamacpp.py serve --model outputs/model-q4_k_m.gguf --port 8081
python scripts/llamacpp.py chat --model outputs/model-q4_k_m.gguf

The deploy command also opens templates/chat_ui.html, a Gaslamp-styled chat WebUI that connects to the local llama-server API.

Troubleshooting Common Errors

1. Out of Memory (OOM) during Training

• Fix 1: Ensure use_gradient_checkpointing = "unsloth" is set in get_peft_model.
• Fix 2: Reduce per_device_train_batch_size to 1 and increase gradient_accumulation_steps (e.g., to 4 or 8).
• Fix 3: Reduce max_seq_length if the task doesn't require long context.
• Fix 4: Switch from LoRA 16-bit to QLoRA 4-bit (load_in_4bit = True).
• Fix 5: Use a smaller pre-quantized model (e.g., 8B instead of 14B).

2. "ValueError: Unsloth requires lora_dropout=0"

• Fix: Unsloth's custom kernels only work if lora_dropout = 0 and bias = "none" in the get_peft_model config.

3. Slow downloading / OOM while loading model

• Fix: Use Unsloth's pre-quantized 4-bit models (e.g., unsloth/llama-3-8b-bnb-4bit). They download 4x faster and fit reliably in RAM.

4. GRPO reward not increasing

• Fix 1: Wait at least 300 steps — GRPO is slow to start.
• Fix 2: Verify your reward function returns correct scores (print intermediate values).
• Fix 3: Use at least 500 rows of training data.
• Fix 4: Ensure the model is ≥1.5B params for generating thinking tokens.
• Fix 5: Try increasing num_generations (e.g., from 4 to 8).

5. Vision training errors

• Fix 1: Ensure remove_unused_columns = False in SFTConfig.
• Fix 2: Use UnslothVisionDataCollator instead of default collator.
• Fix 3: Keep image dimensions between 300-1000px.
• Fix 4: Verify images are accessible (URLs reachable, local files exist).

6. GRPO + vLLM errors

• Fix 1: pip install diffusers if you get a missing module error.
• Fix 2: Update to the latest vLLM version: pip install --upgrade vllm.
• Fix 3: Ensure fast_inference = True is set in from_pretrained().

7. Gradient accumulation bug

• Fix: Use Unsloth's patched trainers. Standard HuggingFace trainers have a known gradient accumulation bug that Unsloth fixes automatically. See: https://unsloth.ai/blog/gradient

8. Hub push failures

• Fix 1: Ensure your HF token has write permissions.
• Fix 2: Set push_to_hub = True and hub_model_id = "username/model-name" in config.
• Fix 3: For private repos, add hub_private_repo = True.

Phase 3: Evaluation (mlx-tune / Apple Silicon)

After training, direct the user to scripts/mlx_eval_template.py. It handles the correct mlx-tune inference API and avoids the common failure modes. Key rules encoded in the template:

1. Load adapter via from_pretrained kwarg — adapter_path="outputs/adapters" passed as **kwargs. Omitting it runs the bare base model silently.
2. Pass raw string to generate — TokenizerWrapper is not callable with return_tensors="pt".
3. Temperature via make_sampler — generate_step has no temperature float; use sampler=make_sampler(temp=0.7).
4. Strip echoed prompt — mlx_lm returns the full sequence including prompt; do raw[len(prompt):].

Run modes:

python ./scripts/mlx_eval_template.py                  # batch
python ./scripts/mlx_eval_template.py --interactive    # REPL
python ./scripts/mlx_eval_template.py --compare        # base vs fine-tuned
python ./scripts/mlx_eval_template.py --style alpaca   # override format

Example Scripts

See the scripts/ directory for ready-to-use templates:

• scripts/unsloth_sft_example.py: Complete SFT training script.
• scripts/unsloth_dpo_example.py: DPO preference training script.
• scripts/unsloth_grpo_example.py: GRPO reinforcement learning script.
• scripts/unsloth_vision_example.py: Vision/multimodal fine-tuning script.
• scripts/mlx_eval_template.py: Evaluation template for Apple Silicon / mlx-tune (batch, interactive, compare modes).
• scripts/setup_colab.py: Auto-setup Unsloth on a Google Colab VM (GPU detection, install, verification).
• scripts/colab_training.py: Helper module for remote Colab training (upload, execute, download, metrics polling).
• scripts/terminal_dashboard.py: Standalone terminal UI using plotext.

Phase 7: Reflection & Memory Synthesis

After the project is complete (post Phase 6 or 6.5), synthesize what was learned into long-term memory at ~/.gaslamp/. This is what makes the agent improve over time.

Step 1 — Extract raw candidates from the completed project:

python3 reflect.py . --extract

Outputs a JSON array to stdout with candidates from gaslamp.md (§ 5 environment, § 6 hyperparameters, § 9 file inventory, § 11 workarounds), memory.md (Discoveries & Notes), and .reflect_hints.json if present. 📖 Learn blocks are skipped — they are generic ML education, not operational lessons.

Inline hints (section: "inline_hints", pre_flagged: true) are already identified as lesson/skill/user — classify them quickly in Step 2 rather than reconstructing from context. Gaslamp.md candidates still require full classification.

reflect.py is a pinned local copy placed by init_project.py — run it from inside the project dir.

Step 2 — Classify and summarize each candidate using your own judgment:

Fast path — inline hints (section: "inline_hints"): if type_hint is already set, accept it and write a ≤120-char body directly without full reconstruction. Only run the full classification pass below if type_hint is absent.

Full classification — for gaslamp.md and memory.md candidates:

• LESSON — an isolated fact or gotcha (→ lessons.md). Write as ≤120-char statement. For § 6 (hyperparameters): extract only non-default values that differed from the obvious choice — skip standard lr/batch entries unless they produced a surprising outcome. For § 9 (file inventory): extract which template was copied and under what name — the "script source → project filename" mapping is the reusable fact.
• USER_PREF — hardware/preference signal (→ user.md). Update or merge with existing.
• SKILL — a connected procedure: synthesize across § 6 + § 9 + § 11 into a scenario recipe with a When: trigger condition (→ skills.md). For § 11 (workarounds): treat each bullet as a separate candidate — do not batch multiple workarounds into one skill. Format:

  When: task=<type> AND hardware=<hw> [AND model_size<=<N>B]
  - Phase N: step one
  - Phase N: step two
  Source: <project_dir>
  

  A recipe earns its place only when it connects multiple cross-section decisions into a reusable procedure. Single isolated facts stay as lessons.

Promotion gate fields — set these on every payload entry (required for gate to function):

• "priority": "high" (write to long-term memory) | "medium" (project-only) | "low" (discard). Promote only facts that will change behaviour on a future project.
• "durability": "durable" (stable across projects) | "recurring" (likely to recur) | "transient" (one-off). Only durable and recurring facts are worth long-term storage.
• "decision": "promote" (write to ~/.gaslamp/) | "keep_project_only" (record in .reflect_decisions.md but do not promote). Use keep_project_only for facts that are specific to this project and unlikely to generalise.
• "source_candidates": list of candidate_id strings from the --extract output (e.g. ["qwen_chip2_sft:section_11:a1b2c3d4"]). Include for traceability.

Entries omitting all four fields are treated as priority=high / durability=durable / decision=promote (v1.1 backwards compat) and will produce a warning. Prefer always including the fields explicitly.

Step 3 — Write to long-term memory:

Write .reflect_payload.json using this schema, then run reflect.py --write:

{
  "user": [
    {
      "title": "Hardware — Apple Silicon",
      "body": "...",
      "date": "YYYY-MM-DD",
      "priority": "high",
      "durability": "durable",
      "decision": "promote",
      "source_candidates": ["project_dir:section_5:a1b2c3d4"]
    }
  ],
  "lessons": [
    {
      "title": "Short title ≤60 chars",
      "body": "One-sentence statement ≤120 chars",
      "source": "project_dir_name",
      "date": "YYYY-MM-DD",
      "priority": "high",
      "durability": "durable",
      "decision": "promote",
      "source_candidates": ["project_dir:section_11:e5f6a7b8"]
    }
  ],
  "skills": [
    {
      "title": "Scenario name",
      "when": "task=<type> AND hardware=<hw> [AND model_size<=<N>B]",
      "steps": ["Phase N: step one", "Phase N: step two"],
      "source": "project_dir_name",
      "date": "YYYY-MM-DD",
      "priority": "high",
      "durability": "durable",
      "decision": "promote",
      "source_candidates": ["project_dir:section_9:c9d0e1f2"]
    }
  ]
}

All three top-level keys are optional — omit any that have no entries. date defaults to today if omitted. Gate fields (priority, durability, decision) are optional but should always be set explicitly.

# Preview gate outcomes and decisions without writing:
python3 reflect.py --write --input .reflect_payload.json --dry-run
# Inspect .reflect_decisions.preview.md before committing
# Then write for real:
python3 reflect.py --write --input .reflect_payload.json

The script handles dedup (sha256), char-limit enforcement (≤3000 chars for lessons/skills, ≤2000 for user), and quarterly archiving (~/.gaslamp/archive/) of evicted oldest entries. After writing, .reflect_decisions.md lists every entry with its outcome (promoted / gate_blocked / dup_skipped / evicted_to_archive).

→ After Phase 7: Update gaslamp.md with a final line in § 11 noting what was reflected: Reflected to ~/.gaslamp/ on YYYY-MM-DD.

Resources

• Unsloth Documentation
• Model Catalog
• Fine-tuning Guide
• RL Guide
• Vision Guide
• TTS Guide
• Saving to GGUF
• vLLM Deployment
• All Notebooks