Imported from TheViziusGroup/vibe-engineering-skills (plugins/ai-and-data/skills/ai-ml-landscape/SKILL.md). Install upstream with npx skills add TheViziusGroup/vibe-engineering-skills --skill ai-ml-landscape. Copyright stays with the author.

AI & Machine Learning Landscape — Practitioner Reference (2026)

The Three-Layer Structure of AI in 2026

1. Frontier model race — fast-moving, narrow quality gaps between top models (~2.7% lead at the top as of March 2026)
2. Production engineering stack — maturing: vLLM/SGLang serving, MCP-standardized agents, contextual-retrieval RAG
3. Classical ML core — stable: gradient boosting still wins most tabular problems

Inference cost collapse: >280-fold reduction from $20.00 to $0.07 per million tokens between Nov 2022 and Oct 2024 (GPT-3.5-equivalent quality, per Stanford HAI). This makes capability cheap and engineering discipline the binding constraint.

Frontier Model Landscape (Mid-2026)

No single model dominates. The top model leads by only ~2.7% (Stanford HAI AI Index, March 2026). The leading closed-weight model's edge over the top open-weight model on Chatbot Arena narrowed from 8.04% (Jan 2024) to 1.70% (Feb 2025).

Key practitioner principle: Build behind an abstraction layer (LiteLLM/OpenRouter) to avoid lock-in. Model names and rankings shift weekly.

Hosted APIs vs Open Weights — Decision Rule

Open-weight options: DeepSeek-V4 (MIT-licensed, 1M context, 1.6T/49B active MoE), Qwen3, Meta Llama 4, Mistral, IBM Granite 4.0.

Threshold to self-host: sustained high QPS where API spend exceeds fully-loaded GPU fleet cost, or a hard data-residency requirement.

Model Selection Strategy (Staged)

1. Start with hosted frontier behind an abstraction layer. Default to strong mid-tier (Claude Sonnet-class, GPT-4.1-mini-class, Gemini Flash-class); escalate to top-tier only for hard tasks.

2. Implement model routing once volume justifies it. A cheap classifier sends easy queries to small/cheap models with confidence-gated escalation. RouteLLM (UC Berkeley/Anyscale/Canva, ICLR 2025) achieved 95% of GPT-4 quality using only ~26% of GPT-4 calls, with up to 85% cost reduction on MT-Bench.

3. Move to open weights self-hosted on vLLM/SGLang when privacy, air-gap, predictable cost at scale, or fine-tuning control becomes the binding constraint.

Build Pattern for LLM Applications

Prompting → RAG → Fine-tuning (in that order)

• Few-shot prompting for behavioral change
• RAG for dynamic/factual/auditable knowledge — use hybrid retrieval (BM25 + dense) + reranking + contextual retrieval
• Fine-tune (LoRA/QLoRA) only for consistent format/style/jargon or cost reduction
• GRPO/DPO only when you have verifiable rewards or preference data and a reasoning/behavior target

Reasoning Models & Test-Time Compute

The o1/o3/R1 paradigm: spending more inference compute on chain-of-thought "thinking tokens" is a complementary scaling axis to parameters/data.

Process Reward Models (PRMs): score intermediate reasoning steps; outperform outcome-only rewards on math (GenPRM: 1.5B model beats GPT-4o on ProcessBench via test-time scaling).

GRPO (Group Relative Policy Optimization): critic-free, group-relative RL with verifiable rewards (RLVR). Dominant post-training method for reasoning. Spawned variants DAPO, Dr. GRPO, GSPO, GMPO.

When reasoning tokens are worth it: genuine multi-step reasoning (math, code generation, legal analysis). Wasteful when wired into pipelines expecting short outputs.

Classical ML — Tabular Data

Gradient-boosted trees remain state-of-the-art on most tabular problems. Deep learning requires more tuning and rarely wins on tabular without genuine unstructured signal.

2025 challenger: TabPFN (transformer tabular foundation model, published in Nature) beats GBDTs on small/medium single tables. XGBoost still wins on large tables.

Practitioner default: GBDT-first (LightGBM → CatBoost → XGBoost depending on data characteristics). Only reach for deep learning with genuine unstructured signal or very large data.

Mathematical Foundations

Linear algebra: tensors are batched multidimensional arrays; matmul/dot products implement every linear layer and attention score; SVD underpins PCA and low-rank adaptation (LoRA literally learns low-rank ΔW). Norms drive regularization (L1 sparsity/Lasso, L2 weight decay/Ridge) and gradient clipping.

Probability: KL/JS/Wasserstein divergences anchor VAEs, GANs, and distribution matching. Cross-entropy is the default classification loss (= MLE under a categorical model). Bias–variance is operationalized by regularization and ensembling.

Optimization: backprop = reverse-mode autodiff applying chain rule over the computational graph. Deep loss landscapes are non-convex but navigable — saddle points (not bad local minima) dominate, and SGD noise helps escape them. AdamW is the safe default; cosine annealing with linear warmup is the standard LR schedule. First-order methods dominate because Hessian-based methods don't scale to billions of parameters.

Classical ML Reference

Regression & Classification

• OLS with Ridge/Lasso/ElasticNet for regularized selection
• GLMs (logistic, Poisson, negative binomial) for non-Gaussian targets
• Gaussian Process Regression for small-data + calibrated uncertainty
• Logistic regression: strong, interpretable baseline — always try it first
• SVMs (kernel trick, RBF): still win on small high-dimensional data

Calibration

Calibration matters in production. Temperature scaling (post-hoc, single-parameter) for neural nets; isotonic/Platt for others. Brier score and reliability diagrams for diagnosis.

Unsupervised & Anomaly Detection

• Clustering: k-means(++)/DBSCAN/HDBSCAN/GMM/spectral
• Dimensionality reduction: PCA/UMAP/t-SNE (UMAP preferred for preserving global structure and speed)
• Anomaly detection: isolation forest/LOF/autoencoders

Model Selection

• Stratified/group/time-series CV to prevent leakage
• Metric choice: AUC-PR for imbalanced, MCC as balanced single number, log-loss for probabilistic
• Optuna (TPE/Bayesian) + Hyperband/ASHA for hyperparameter tuning
• Leakage is the silent killer: target leakage, train/test contamination via preprocessing, and temporal leakage cause the most "great offline, broken online" failures

Deep Learning Foundations

Activations & Normalization

• ReLU: default for CNNs
• GELU/SiLU(Swish): dominate transformers (smooth, work well with normalization)
• BatchNorm: CNNs
• LayerNorm/RMSNorm: transformers (RMSNorm is cheaper, now standard in LLMs)
• Pre-norm (LayerNorm before sublayer): dominates modern transformers for stability at depth

Initialization & Precision

• He/Kaiming for ReLU nets; Xavier for tanh; orthogonal for RNNs
• BF16: LLM-training default (wider dynamic range than FP16, no loss scaling)
• FP8: production on Hopper/Blackwell (DeepSeek-V3 trained in FP8)
• Mixed precision memory tools: gradient checkpointing, gradient accumulation, activation recomputation

Residual Connections

Enable very deep networks by giving gradients an identity path.

The Transformer & Attention

Architecture types:

• Encoder-only (BERT): classification/embedding
• Encoder-decoder (T5): seq2seq tasks
• Decoder-only (GPT): dominates generative LLMs

Attention components:

• Scaled dot-product attention over Q,K,V
• Multi-head attention
• RoPE (rotary positional encoding): now standard; YaRN/position interpolation extends context
• FlashAttention (v1→v3): IO-aware, exact (not approximate), tiling attention in SRAM — mitigates O(n²) attention cost
• KV cache: makes autoregressive decoding tractable
• FFN uses SwiGLU

Mixture of Experts (MoE): dominant frontier-scaling lever. Sparse gating + routing + load balancing. DeepSeek-V3 activates 37B of 671B params per token. Mixtral, DeepSeek-MoE, Qwen3-MoE all use this architecture.

LLM Training & Post-Training

Tokenization

• BPE/WordPiece/SentencePiece/tiktoken; larger vocabulary = shorter sequences but larger embedding tables
• Chinchilla scaling (~20:1 tokens:params) guided compute-optimal training; inference-cost economics now push toward "overtrain a smaller model"

Pre-training Infrastructure

• Curated/deduplicated/quality-filtered corpora + heavy synthetic data (Phi "textbooks" thesis)
• Tensor/pipeline/data parallelism; ZeRO/DeepSpeed; FSDP; Megatron-LM

Alignment Pipeline

1. SFT (Supervised Fine-Tuning): high-quality instruction examples; "LIMA / Less is More" — a few thousand high-quality examples often beat massive noisy sets
2. DPO (Direct Preference Optimization): reference-model based, no explicit reward model; simple default
3. GRPO: critic-free, group-relative, RLVR; dominates reasoning post-training

PEFT (Parameter-Efficient Fine-Tuning)

• LoRA: low-rank adapters; workhorses of production fine-tuning
• QLoRA: 4-bit NF4 base + LoRA; enables fine-tuning large models on consumer hardware
• DoRA/VeRA: refinements of LoRA

LLM Inference Optimization

Quantization Formats

Rule: avoid INT4 for math/code/reasoning-critical paths.

Serving Engines

Continuous batching + disaggregated prefill/decode (separate compute-bound prefill and bandwidth-bound decode pools) are the 2026 production patterns.

Speculative decoding (draft+verify, Medusa/EAGLE): 2–4× latency wins losslessly.

Evaluation & Benchmark Skepticism

The contamination problem is severe. A single model can swing 17–35 points between SWE-bench Verified and the standardized-scaffold Pro leaderboard — scaffold and harness move scores more than model swaps.

Key benchmarks:

• MMLU(-Pro), GPQA Diamond: knowledge/reasoning
• HumanEval/MBPP, SWE-bench Verified/Pro: coding
• GSM8K/MATH: math reasoning
• MMMU: multimodal
• LiveBench/LiveCodeBench: contamination-resistant
• Chatbot Arena Elo: human preference
• RULER/HELMET: long-context
• TruthfulQA/FActScore: factuality
• RAGAS: RAG evaluation

Decision rule for benchmarks:

• Never select on a single leaderboard number
• Build a 50–200 case internal eval of your real workflows
• Weight SWE-bench Pro (standardized scaffold) over Verified for coding
• When a benchmark and your production results diverge sharply, trust contamination/scaffold effects over the leaderboard
• LLM-as-judge is widespread but carries position/verbosity/self-preference biases

RAG Fundamentals (Quick Reference)

Production default (2026): hybrid search (BM25 + dense via RRF) + cross-encoder reranking + parent-child chunking.

Anthropic Contextual Retrieval (Sept 2024): prepend LLM-generated chunk summary before embedding/indexing.

• Contextual Embeddings alone: 35% reduction in retrieval failure (5.7%→3.7%)
• • Contextual BM25: 49% reduction (→2.9%)
• • Reranking: 67% reduction (→1.9%)

RAG vs fine-tuning vs long-context:

• RAG: dynamic/proprietary knowledge with citations
• Fine-tuning: changing behavior/format/tone/domain style
• Long-context: single-document deep reasoning where doc fits
• They combine

Agent Stack & MCP

Model Context Protocol (MCP): Anthropic, Nov 2024; donated to Linux Foundation Dec 2025; adopted by OpenAI, Google, Microsoft, Amazon. "USB-C for AI tools." Over 97M monthly SDK downloads and 10,000 active servers at donation time.

Production failure modes: runaway loops, unbounded retries, silent context truncation. Iteration limits, cost ceilings, and termination conditions are mandatory, not polish.

Single-agent suffices for ~80% of cases. Multi-agent adds cost and non-determinism.

Framework selection:

• LangGraph: production default for regulated/auditable workflows; graph state machines with checkpointing
• CrewAI: fastest role-based multi-agent prototyping
• Pydantic AI: typed, minimal boilerplate

Generative AI Across Modalities

Image

• GANs and VAEs gave way to diffusion (DDPM→DDIM→Latent Diffusion = Stable Diffusion)
• Now flow matching / rectified flow (backbone of SD3, FLUX)
• Classifier-free guidance, ControlNet (pose/depth/edge conditioning), LoRA/DreamBooth personalization are standard
• FLUX (Black Forest Labs): open-weight image generation leader in 2026

Video

• Standardized on diffusion transformers (DiT) over spacetime patches (Sora, Veo, Kling, Seedance, WAN, Hunyuan)
• By early 2026: 8–25s clips at native resolution with synchronized audio and plausible physics
• Weaknesses: long-range temporal coherence and quadratic attention cost

Audio

• TTS: Tacotron→FastSpeech→VITS→XTTS/F5-TTS/Kokoro; ElevenLabs leads commercial quality
• ASR: Whisper dominates
• Discrete audio tokens: EnCodec/DAC

Multimodal/VLMs

• Vision encoders trend SigLIP over CLIP, with tile-based high-res
• Open options: LLaVA, Qwen-VL, InternVL, Molmo, Pixtral

MLOps & Infrastructure

Hardware

Frameworks

• PyTorch: dominant (eager + torch.compile/Inductor)
• JAX: TPU/research
• Hugging Face stack: Transformers, Diffusers, PEFT, TRL, Datasets, Accelerate — the default
• Experiment tracking: W&B (research favorite), MLflow (open, ubiquitous)

Observability & Monitoring

• LLM serving/observability: LangSmith, Langfuse, Arize Phoenix, Helicone, Braintrust, W&B Weave
• Drift/monitoring: Evidently, NannyML, Arize, WhyLabs, Fiddler; PSI/KS tests; distinguish covariate/label/concept shift
• Explainability: SHAP/LIME/Integrated Gradients (caution: attention weights ≠ explanations)

Advanced Architectures

State-Space Models (SSMs)

• Mamba (selective state spaces, Mamba-2): linear complexity, ~5× throughput
• Critical weakness: pure SSMs fail at retrieval/in-context copying — removing attention layers drops retrieval accuracy to near-zero
• Hybrids win in production: Jamba (AI21), Zamba2, NVIDIA Nemotron-H, IBM Granite 4.0 — small fraction of attention layers interleaved with Mamba blocks

Long Context

• 1M+ token windows are table stakes across flagships
• "Lost in the middle" and needle-in-haystack failures persist
• RAG vs long-context is a cost/freshness/auditability decision, not a pure capability one

Reasoning Research

• Test-time compute scaling via PRMs
• MCTS for reasoning
• Self-consistency voting
• Formal-verification systems (AlphaProof/AlphaGeometry 2)

Safety, Alignment & Governance

Alignment Approaches

• Constitutional AI / RLAIF (Anthropic): AI self-critique for alignment
• Mechanistic interpretability: sparse autoencoders decompose superposed/polysemantic activations; circuit tracing with cross-layer transcoders lets researchers trace and intervene on causal pathways (Anthropic, open-sourced May 2025; MIT Tech Review 2026 Breakthrough Technology)
• Outer alignment (specification) vs inner alignment (goal generalization)
• Reward hacking and Goodhart's Law: recurring failure modes

EU AI Act Timeline (Critical Deadlines)

Fines: up to €35M or 7% of global turnover (prohibited practices) — higher ceiling than GDPR. The voluntary GPAI Code of Practice offers a "presumption of conformity" safe harbor.

Action: If you touch EU users, classify your system now against the AI Act risk tiers. If you fine-tune a GPAI model substantially, you may become a "provider" with heavier obligations.

Fairness

Demographic parity vs equalized odds vs individual fairness are mutually incompatible (impossibility theorems). Tools: Fairlearn, AI Fairness 360.

Privacy

Differential privacy, federated learning, and defenses against membership-inference/model-inversion/poisoning attacks.

US Framework

NIST AI RMF (Govern/Map/Measure/Manage) is the US reference standard.

Durable Principles (Outlast Model Names)

1. Baseline-first: always establish a simple baseline before reaching for complex architectures
2. Data quality over architecture: garbage in, garbage out — no model overcomes bad data
3. Route, don't over-provision: tiered model fleets beat one expensive frontier model
4. RAG before fine-tune: try prompting and retrieval before committing to training
5. Guardrails and observability before launch: build I/O filtering, tracing, and cost monitoring into the initial design
6. Benchmark skepticism: build internal evals on your real workflows; never select on a single leaderboard number
7. Open vs closed is a cost/privacy/capability trade-off, not an ideology: evaluate quarterly as quality gaps narrow