experiment

Imported from ruihangdu/CS372_Final_Project (experiment/AGENTS.md). Install upstream with npx skills add ruihangdu/CS372_Final_Project --skill experiment. Copyright stays with the author.

CS372 MLLM Content Moderation Evaluation — Agent Instructions

This is the master reference for all agents working in ~/CS372/experiment/. Each subdirectory has its own AGENTS.md with component-specific instructions. Read this file first, then the relevant subdirectory's file.

Note: project_proposal/AGENTS.md was for the proposal-writing phase only and is not relevant to experiment/. These files supersede it.

Project Identity

This is a CS372 research project evaluating how well frontier Multimodal LLMs classify YouTube video clips by age-appropriateness (0-4 / 5-8 / 9-12 / 13-17 / 18+) according to YouTube's actual Community Guidelines and YouTube Kids content policies.

Dataset: MMOB (Multimodal Malicious or Benign) — 1,903 mp4 video clips (~10s each) from YouTube children's content. Source paper: arXiv:2405.06128.

Dual-Directory Architecture

There are two copies of this experiment. Understand the difference:

Rules:

• Source code changes: edit files in ~/CS372/experiment/, then copy to /scratch if needed for execution
• Data artifacts (frames, audio, video, transcripts): only exist on /scratch, never committed to git
• All hardcoded paths in scripts point to /scratch. This is intentional
• The .gitignore excludes .env, *.mp4, *.wav, *.jpg, *.png, logs/, __pycache__/

Related data directories on /scratch (outside experiment/):

• /scratch/.../cs372_research/mob_video/MOB_RUN/clips/video/{benign|malicious}/ — original MMOB video clips
• /scratch/.../cs372_research/mob_video/MOB_RUN/frames/video/{benign|malicious}/ — original 25fps frame extractions
• /scratch/.../cs372_research/mob_audio/all_audio/{videoID}/ — original MMOB audio (.wav)
• /scratch/.../cs372_research/mmob_repo/mob.csv — MMOB metadata CSV (1,875 rows)

Environment

Python interpreter (required — system Python 3.12 lacks necessary packages):

/software/spack/opt/spack/linux-x86_64_v3/python-3.14.2-h3pybvsxurte2fyoiy7n7d57dvntmfr5/bin/python3

Key Python packages available: httpx, pandas, numpy, Pillow, python-dotenv, faster_whisper, imageio_ffmpeg, tqdm

Git: Standard git push works (SSH config is correctly configured).

Slurm (Stanford FarmShare): GPU nodes available (6x oat nodes, 4x NVIDIA L40S each) but CUDA toolkit libs are NOT installed. CPU Whisper works fine. Details: /scratch/.../cs372_research/slurm_gpu_findings.md

API Keys

Stored in /scratch/users/ruihangd/opencode_work/cs372_research/experiment/config/.env:

Rules: Never log, print, or write API keys to output files. Load via python-dotenv only.

Age Tiers

Used throughout all scripts. Ordinal mapping for MAE calculation:

Critical Design Decisions (DO NOT CHANGE)

1. No "Tone & Emotional Impact" category in prompts or checklists. This is a deliberate "policy gap trap" — models that over-restrict scary-but-not-violent content are hallucinating policy rules. Step 2 verification should catch this. This is a core research contribution.

2. MMOB binary labels (malicious/benign) are NOT ground truth. They do not align with YouTube's policy framework. Our Gemini 3 Flash checklist labels (corrected by human review) are the ground truth.

3. CSV uses hyphens, filesystem uses underscores for YouTube video IDs. The data_prep/id_mapping.py script handles this mapping. The clip_manifest.csv uses filesystem-style IDs (underscores) as clip_id.

4. "Explore" setting for 9-12 tier, not "Older". This matches YouTube's actual nomenclature.

5. Severity-to-age-tier mapping is deterministic. Defined in SEVERITY_TO_AGE dict in labeling/run_labeling.py. The MAX severity across all 7 categories determines the final tier.

Project Status

Phase 0: Environment Setup

• Directory structure created on both ~/CS372/ and /scratch
• Config file (config/config.py) with all paths and parameters
• API keys configured in /scratch/.../config/.env
• Policy text frozen at prompts/policy_text/youtube_kids_policies.txt
• System prompts written for Condition A and Condition B

Phase 1: Data Preparation

• 1A: ID mapping — clip_manifest.csv (1,903 rows, resolves hyphen/underscore)
• 1A: Video-audio muxing — 1,902 muxed mp4s at data_prep/muxed_clips/
• 1B: Frame sampling at 1fps — 1,396 clip dirs at data_prep/sampled_frames/
• 1B: Frame resampling at 5fps (200 pilot clips) at data_prep/sampled_frames_5fps/
• 1B: Frame resampling at 2fps (200 pilot clips) at data_prep/sampled_frames_2fps/
• 1C: Whisper transcription of 200 pilot clips — 375 JSON files at data_prep/transcripts/
• 1B-ext: 5fps frame sampling for ALL ~1,875 clips (currently pilot-only)
• 1C-ext: Whisper transcription for remaining ~1,700 clips

Phase 2: Pilot Sampling

• 200 clips selected, stratified: 80 (v0_a0) / 50 (v1_a1) / 40 (v1_a0) / 30 (v0_a1)
• All 200 have frames + transcript + muxed video

Phase 3: Step 1 Baseline Pilot

• Condition A: GPT-4o (200 clips, 0 errors)
• Condition A: Claude 3.5 Sonnet (200 clips, 0 errors)
• Condition A: Gemini 2.0 Flash (200 clips, 0 errors)
• Condition B: Gemini 2.0 Flash (200 clips, 0 errors)
• Condition A analysis report (results/pilot/condition_a_analysis.md)

Phase A: Full-Dataset Labeling

• Gemini 3 Flash checklist pipeline — 1,873 clips labeled, 1 error (QWheNtS5kP0_3)
• Results at results/labels/labels_full.jsonl
• Distribution: 883 (0-4), 171 (5-8), 268 (9-12), 292 (13-17), 259 (18+)
• 63.2% high uncertainty (maxed 3 verification rounds)
• Re-label failed clip QWheNtS5kP0_3

Phase B: Human Review & Policy Refinement (IN PROGRESS)

• Review queue generated — 150 clips
• User manual review of 10 test clips (M0.5d + M0.5f) — identified 30% override rate
• Root cause analysis: policy ambiguities in Violence ("non-scary") and Weapons (gaming context)
• M0.5g: Designed two labeling approaches for 20-clip pilot test
  • Approach A: Natural Language Rationale (simple, validated approach)
  • Approach B: Multi-Axis Structured (explicit sub-reasoning for edge cases)
• M0.5g-impl: Implemented both labeling systems
• M0.5h: Ran pilot test on 20 clips (40 labels total)
  • Approach A: 100% ACCEPT rate, avg 1.15 verify rounds, $0.036 cost
  • Approach B: 90% ACCEPT rate, avg 1.45 verify rounds, $0.043 cost
• M0.5h-web: Built local web review interface (http://localhost:5000)
• M0.5i: User comparative review of 40 labels → choose winning approach
• M0.6: Run winning approach on 500-clip stratified sample
• M0.8-M0.9: Sample 50 clips from 500 for final quality review

Phase C: Evaluation (COMPLETE — RESULTS IN ANALYSIS DIRECTORY)

The 5 Final Experiments (500-clip stratified sample, seed=42): All experiments use GPT-5.1, Gemini 2.5 Flash, and Claude 4.5 Sonnet (3 models each, except Exp 3 which is Gemini-only).

Quick Reference — Run All 5 on 500-clip sample (Completed):

# Prerequisite (completed)
python experiment/data_prep/build_sample_500.py

# Exp 1: Zero-shot Baseline (completed)
python experiment/step1_baseline/run_baseline_step1.py --model gpt51 --subset data_prep/selected_500_ids.csv
python experiment/step1_baseline/run_baseline_step1.py --model gemini25flash --subset data_prep/selected_500_ids.csv
python experiment/step1_baseline/run_baseline_step1.py --model claude45sonnet --subset data_prep/selected_500_ids.csv

# Exp 2: Zero-shot Multi-Axis Frames (completed)
python experiment/step1_baseline/run_multiaxis_step1.py --model gpt51 --subset data_prep/selected_500_ids.csv
python experiment/step1_baseline/run_multiaxis_step1.py --model gemini25flash --subset data_prep/selected_500_ids.csv
python experiment/step1_baseline/run_multiaxis_step1.py --model claude45sonnet --subset data_prep/selected_500_ids.csv

# Exp 3: Multi-Axis Video (Gemini only) (completed)
python experiment/step1_baseline/run_multiaxis_video_step1.py --subset data_prep/selected_500_ids.csv

# Exp 4: Naive Labeler+Verifier Loop (completed)
python experiment/step2_verification/run_phase_c_step2.py --model gpt51 --subset data_prep/selected_500_ids.csv
python experiment/step2_verification/run_phase_c_step2.py --model gemini25flash --subset data_prep/selected_500_ids.csv
python experiment/step2_verification/run_phase_c_step2.py --model claude45sonnet --subset data_prep/selected_500_ids.csv

# Exp 5: CRIT 4-Stage Dialogue (both as labeler and verifier) (completed)
python experiment/step3_crit_verifier/run_crit_step2.py --labeler-model gpt51 --verifier-model gpt51 --subset data_prep/selected_500_ids.csv
python experiment/step3_crit_verifier/run_crit_step2.py --labeler-model gemini25flash --verifier-model gemini25flash --subset data_prep/selected_500_ids.csv
python experiment/step3_crit_verifier/run_crit_step2.py --labeler-model claude45sonnet --verifier-model claude45sonnet --subset data_prep/selected_500_ids.csv

# Review UI (once all results are generated)
bash experiment/review_ui_v4/start_server.sh
# Then open http://localhost:5000

Key outputs:

• All results written to results/full/*.jsonl
• Review state in results/review_queue/v4_review_results.jsonl + v4_review_state.json
• Full analysis and evaluation reports written to experiment/analysis/

For full details on each experiment, see:

• Exp 1-2: step1_baseline/AGENTS.md

• Exp 3: step1_baseline/AGENTS.md (subsection "Exp 3: Multi-Axis Native Video")

• Exp 4: step2_verification/AGENTS.md

• Exp 5: step3_crit_verifier/AGENTS.md

• Full-dataset Whisper transcription (M1.1-M1.3) — 1,902 clips transcribed

• M1.4: Transcript quality spot-check — PASS (100% success rate)

• Phase C runners implemented and fixed (M2.3-M2.5):

  • step1_baseline/run_phase_c_step1.py — with --subset flag and no_frames error record
  • step1_baseline/run_baseline_step1.py — NEW: baseline runner for old models
  • step2_verification/run_phase_c_step2.py — with --subset flag, rounds counter fixed
  • step3_crit_verifier/run_crit_step2.py — parallel 4-stage CRIT (Definition→Elenchus→Dialectic→Maieutics)
  • data_prep/build_sample_500.py — NEW: generates selected_500_ids.csv

• Run build_sample_500.py to generate selected_500_ids.csv (5-minute one-time setup)

• CRIT 20-clip sanity check + cost estimate

• Experiments 1-5: full 500-clip runs completely generated

• Human golden label review (tiered: auto-accept / quick-confirm / full-review)

• Analysis: accuracy, MAE, confusion matrices, policy gap trap, convergence (completed in experiment/analysis)

• Publication-quality figures and reports

Git

• Commit 7745e51: "Checkpoint: Phase A labeling complete, review queue generated"
• Pushed to git@github.com:ruihangdu/CS372_Final_Project.git (master)

Subdirectory Reference