Imported from r0b0tlab/DeepSeek-V4-Flash-DSpark-v026-SM121 (AGENTS.md). Install upstream with npx skills add r0b0tlab/DeepSeek-V4-Flash-DSpark-v026-SM121. Copyright stays with the author.

AGENTS.md — coding / ops agents working this repo

Name: DeepSeek-V4-Flash-DSpark-v026-SM121
Author: r0b0tlab

This file is the ready-to-run contract for Hermes, Cursor, and human operators.

What this repo is

Dual-GB10 / SM121 serving package for deepseek-ai/DeepSeek-V4-Flash-0731.
Wrapper + profiles + gates around frozen image dspark-r0b0tlab:production-v026-b12x-native-v11.
Not a model weight redistributor. Weights stay on local/HF paths.
Not r0b0bench itself — systems benchmarks live in the separate r0b0bench client repo (use v1.0.0rc2+).

Optimal engine configs (canonical)

A) Production (decode / concurrency / BFCL / default serving)

Profile: profiles/dspark-r0b0tlab-production.env

Knob	Value	Why
image	`…-native-v11` / `sha256:2dfe24a3…`	frozen lossless regular graphs
cudagraph	regular + strict	only dual-rank proven path
compile	NONE (`mode:0`)	compile-on garbles DSV4 on this stack
cudagraph_mode	FULL_DECODE_ONLY	matches regular wrapper
max_model_len	327680	production context pin
max_num_seqs	16	concurrency ladder / BFCL
max_num_batched_tokens	16384	production prefill chunk
gpu_memory_utilization	0.835	fixed admission
kv_cache_memory_bytes	15998753178 (~14.9 GiB)	fixed; ~156k token pool at production seqs
K (MTP)	6	dspark draft
KV dtype	nvfp4_ds_mla	production KV
MoE	flashinfer_b12x	B12X

B) Max-context / 1M NIAH (capacity only)

Profile: profiles/dspark-r0b0tlab-1m.env

Knob	Value	Why
max_model_len	1048576	full 1M advertised window
max_num_seqs	1	C1 capacity
max_num_batched_tokens	8192	required so auto-KV still holds ≥1M tokens
gpu_memory_utilization	0.88	more room for KV
kv_cache_memory_bytes	auto	do not pin 16 GiB
everything else	same as production regular NONE+FDO

Do not raise long-1m batched to 16384/32768 on dual-GB10:

16384 → ~642k KV tokens (90% of 1M fails capacity)
32768 → engine fails start (activation steals KV; needs ~31 GiB with ~16 GiB free)

C) Breakable A/B only

Profile: profiles/dspark-r0b0tlab-breakable-ab.env
PIECEWISE + breakable. Not production default. DFlash draft may run eager under piecewise.

Immutable identity (fail closed)

image_tag        dspark-r0b0tlab:production-v026-b12x-native-v11
image_id         sha256:2dfe24a303743722046c55bf2cf3ef9c9e6a40a008f73be16f9aa5c2ca38cf9b
package          0.26.0+dspark.sm121.3
integrated_vllm  fbad93432ebf60c8345397b3032fe03bf5d239cc
oci_source_rev   cc6b2be9baa75d227f4f54d45bbbc98fb5b64094
model_revision   9e165c30e2704aec5d9d593cce3eebd58bbef1cb
runtime_manifest recipe/runtime-manifest.vllm-sm121.3.json
manifest_sha256  f6777b31e8c8977361d629ca99c37a199d3b52b9abc134f1de624eb202f083a9

CompilationConfig JSON (production / long-1m):

{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_implementation":"regular","cudagraph_strict":true}

Ready-to-run (minimum path)

0) Prerequisites

Two GB10 nodes, IB/RDMA, Docker, SSH head→worker.
Model weights at the path your launcher mounts as /model (see scripts/run-dspark-dual-gb10.sh).
Image loaded identically on both ranks (docker load of the same image_id).
Clean wrapper git tree (launcher fail-closes on dirty).
Optional systems client: clone/install r0b0bench 1.0.0rc2+ with [bfcl] extra and DSV4 BFCL registry scripts.

1) Launch production

cd /path/to/this/repo
export RUNTIME_AUDIT_MANIFEST="$PWD/recipe/runtime-manifest.vllm-sm121.3.json"
set -a && source profiles/dspark-r0b0tlab-production.env && set +a
# point manifest if profile still references an artifacts path:
export RUNTIME_AUDIT_MANIFEST="$PWD/recipe/runtime-manifest.vllm-sm121.3.json"
export DSPARK_RUNTIME_AUDIT_MANIFEST="$RUNTIME_AUDIT_MANIFEST"
MASTER_ADDR=<head-ip> WORKER_HOST=<worker-ip> \
  scripts/run-dspark-dual-gb10.sh

Wait until curl -sf http://<head>:8888/v1/models succeeds (often 12–20 minutes cold).

2) Runtime gate (both ranks)

python3 scripts/runtime_gate.py \
  --container dspark_vllm \
  --worker-host <worker-ip> \
  --expected-implementation regular \
  --expected-strict \
  --expected-cudagraph-mode FULL_DECODE_ONLY \
  --expected-compilation-config '{"cudagraph_implementation":"regular","cudagraph_strict":true}' \
  --base-url http://<head>:8888 \
  --model deepseek-v4-flash-dspark \
  --out /tmp/runtime-gate.json

Require status=PASS, errors=[], schema ≥3.

3) In-repo reproducibility suite

OUT=/tmp/dspark-repro-$(date -u +%Y%m%dT%H%M%SZ)
scripts/run-repro-suite.sh \
  --base-url http://<head>:8888 \
  --model deepseek-v4-flash-dspark \
  --worker-host <worker-ip> \
  --out "$OUT"

4) Systems package (r0b0bench rc2+) — production profile

# separate checkout
cd /path/to/r0b0bench && pip install -e '.[bfcl]'
export R0B0BENCH_AQUILA_SCRIPTS=/path/to/dsv4-bfcl-scripts   # served-id registry
export R0B0BENCH_BFCL_PYTHON=$PWD/venv/bin/python
export BFCL_NUM_THREADS=4 BFCL_HTTP_TIMEOUT=3600 BFCL_MAX_TOKENS=8192

r0b0bench run --profile systems \
  --base-url http://<head>:8888/v1 \
  --model deepseek-v4-flash-dspark \
  --tokenizer /path/to/DeepSeek-V4-Flash-0731/hf \
  --output /tmp/r0b0bench-out \
  --only canary,bfcl_mt,bfcl_ast,latency,concurrency,throughput

Systems lanes (standard package):
canary → bfcl_mt → bfcl_ast → latency → concurrency → throughput → niah

For production runs, prefer skipping niah here and running max-context NIAH on long-1m (below). Production physical KV (~156k tokens with fixed pin) cannot hold 25/50/90% of 327k.

5) Max-context NIAH (long-1m)

# stop production containers first (single engine)
set -a && source profiles/dspark-r0b0tlab-1m.env && set +a
export RUNTIME_AUDIT_MANIFEST="$PWD/recipe/runtime-manifest.vllm-sm121.3.json"
export DSPARK_RUNTIME_AUDIT_MANIFEST="$RUNTIME_AUDIT_MANIFEST"
export VLLM_ENGINE_ITERATION_TIMEOUT_S=3600 VLLM_RPC_TIMEOUT=3600
MASTER_ADDR=<head-ip> WORKER_HOST=<worker-ip> scripts/run-dspark-dual-gb10.sh

# after READY — depths always 25/50/90% of (max_model_len-64)
# client timeout for 90% must be >> 2h (recommend 12h)
scripts/run-niah-max-context.py \
  --base-url http://<head>:8888 \
  --model deepseek-v4-flash-dspark \
  --tokenizer /path/to/hf/tokenizer.json \
  --timeout-s 43200 \
  --out /tmp/niah-1m

6) One-shot option-2 orchestrator (this machine’s layout)

scripts/run-option2-pubsuite.sh \
  --head <head-ip> --worker <worker-ip> \
  --out /tmp/option2-pubsuite

Order: wait/finish long-1m NIAH → production launch → r0b0bench systems (no niah) → repro → publish/OPTION2_PACK.md.

Hard rules for agents

Never claim release from compile logs, container start, or overlay unit tests alone.
Evidence outside git checkout — gates, benches, BFCL projects, NIAH prompts.
Zero infra failures in publishable rows; count model failures separately.
Do not mix breakable vs regular rows, or vllm-bench vs openai_portable rows.
NIAH depths = fractions of advertised max_model_len from /v1/models (max-context test). Record kv_cache_size_tokens when physical KV < advertised.
torch.compile off unless user explicitly opts into experimental diagnostic.
Live Hermes install/config/gateway changes need explicit user OK; repo work does not.
Dirty wrapper tree / mismatched image IDs / wrong model revision → fail closed, do not “force launch”.
Quality pillars (full QA/IFEval/HE/GSM8K) may be NOT_IMPLEMENTED in r0b0bench RC — label honestly; do not wash scores.
Stop means stop. Do not restart a healthy multi-hour NIAH/BFCL job to “optimize” without user direction.

Definition of done (agent task)

Production profile launches; both ranks same image_id
runtime_gate.py PASS (errors=[])
Repro suite artifacts under $OUT/repro/
r0b0bench report.json for systems lanes (or explicit NOT_IMPLEMENTED)
Max-context NIAH results with per-depth PASS/FAIL/ERROR + elapsed
Publication pack lists SHAs, configs, and invalid_for_publish honestly
No secrets or private absolute paths committed

Key paths

Path	Role
`profiles/*.env`	fail-closed launch knobs
`scripts/run-dspark-dual-gb10.sh`	dual-rank launcher
`scripts/runtime_gate.py`	both-rank identity + markers
`scripts/run-repro-suite.sh`	in-repo reproducibility
`scripts/run-niah-max-context.py`	25/50/90 NIAH
`scripts/run-option2-pubsuite.sh`	full systems+repro+1M pack
`benchmarks/`	correctness, staggered, concurrent, gsm8k
`recipe/runtime-manifest.vllm-sm121.3.json`	audit pin
`docs/REPRODUCIBILITY.md`	detailed reproduce notes
`docs/NATIVE_V11_FREEZE.md`	freeze record

Rollback

set -a && source profiles/dspark-r0b0tlab-breakable-ab.env && set +a
MASTER_ADDR=<head> WORKER_HOST=<worker> scripts/run-dspark-dual-gb10.sh

Same image ID. Do not rebuild to roll back.

DeepSeek-V4-Flash-DSpark-v026-SM121

About