Imported from Aress07/Hierarchical-DRL-for-Financial-Trading (AGENTS.md). Install upstream with npx skills add Aress07/Hierarchical-DRL-for-Financial-Trading. Copyright stays with the author.

Project Documentation

Project Goal

Compare 2-level (L1-L4) vs 3-level (L1-L2-L3) hierarchical DRL architectures for trading, plus ablation studies (single-level PPO, component analysis) and macro feature expansion.

Reference: docs/2026-06-01/EXECUTION_PLAN.md (Phase A/B/C master plan).

Final comparison results: docs/2026-06-01/COMPARISON_RESULTS.md

Architecture Overview

2-Level Architecture

Level 1 (Trend Agent)     -> 3-action discrete (bear/sideways/bull)
Level 4 (Merged Agent)    -> 9-action discrete (direction x size combos)

L1 predicts market regime; L4 jointly predicts direction + sizing.

3-Level Architecture

Level 1 (Trend Agent)     -> 3-action discrete (bear/sideways/bull)
Level 2 (Action Agent)    -> 3-action discrete (sell/hold/buy)
Level 3 (Sizing Agent)    -> 5-action discrete [0%, 25%, 50%, 75%, 100%]

L1-L2-L3 pipeline. L3 changed from SAC Box(0,1) to PPO Discrete(5). L2 reward has 10x consistency bonus, holding bonus, entropy bonus.

Single-Level PPO (New)

PPO Agent                 -> 9-action discrete (same as L4, no hierarchy)

Flat agent with no L1/L2 conditioning. Tests whether hierarchy adds value.

Current Status (June 1, 2026) — Post-Comparison

Walk-Forward Results (3 folds, 5 seeds each)

Holdout Results (QQQ, IWM, ETH-USD)

Neither architecture beats buy-and-hold. See docs/2026-06-01/COMPARISON_RESULTS.md.

Code Changes Complete (Phase A)

• train_single_level.py: Flat PPO Discrete(9) agent, strips conditioning dims, no consistency bonus
• Ablation flags: --no-l1, --no-turnover, --no-vix on train_level4.py; --no-l1 on train_level3.py
• Macro features: 3 FRED series (DGS10, DFF, BAA-AAA spread), N_FEATURES=23, requires FRED_API_KEY env var
• reward.py: gamma param in turnover_penalty and compute_level4_reward; use_consistency_bonus flag
• trading_env.py: reward_gamma and use_consistency_bonus constructor params

Overfitting Mitigation (Phase 1–2 Complete, June 2)

• Phase 1 (quick wins): n_epochs 10→3, dropout 0.0/0.1→0.3, model sizes reduced (LSTM 128→64, Transformer d_model 64→32), weight_decay 1e-4, EvalCallback with 80/20 train/val split, best model loading after training
• Phase 2 (data augmentation): src/environment/augmentation.py — NoiseAugmentation, TimeWarpAugmentation, RegimePerturbation wrappers; --augmentation flag on all 5 training scripts; validation envs skip augmentation for clean eval signal
• All 41 tests pass.

Complete (Phase B — GPU Training on Lightning AI) ✓

• 4 ablation experiments (fold 2, SPY) — all completed
• Single-level PPO training (fold 2, SPY) — completed
• Full per-fold retraining with macro features (3 folds, 5 seeds, both archs) — completed
• Holdout evaluation (QQQ, IWM, ETH-USD) — completed

Key Findings (Ensemble Phase Complete, June 4)

1. Previous Sharpe 1.7-2.3 was entirely from data leakage (trained 2015-2021, tested on 2018-2021 overlap).
2. Proper per-fold walk-forward: neither arch beats B&H. 3-level marginally better.
3. L2 directional accuracy (~48%) remains the structural bottleneck.
4. Ensemble of 5 seeds reduces std by 37–56% on 2-level (all 3 folds have 5 seeds). 3-level ensemble limited by 1-seed coverage on folds 1–2.
5. Ensemble improves mean Sharpe modestly (+0.14 on SPY 2-level, +0.07 on SPY 3-level) but not enough to beat B&H.
6. Overfitting is from weak features, not model capacity. Ensembling fixes seed variance but cannot fix feature signal.
7. Merged 2-level (L4) is more susceptible to overfitting than 3-level pipeline.

Action Spaces Per Level

Reward Functions

Key File Locations

src/
  environment/
    trading_env.py       # Core env with reward_gamma/use_consistency_bonus params
    reward.py            # gamma param in turnover_penalty, compute_level4_reward
    portfolio.py         # Cash/holdings tracking
    wrappers.py          # Level2/3/4 conditioning wrappers
    risk_management.py   # Vol cap, drawdown stop, cool-off
    ood_detection.py     # Mahalanobis OOD detection
  agents/
    level1_trend_agent.py
    level2_action_agent.py
    level3_sizing_agent.py   # PPO Discrete(5)
    level4_merged_agent.py
  training/
    pretrain_level1.py    # Transformer L1, fold-aware
    train_level2.py       # PPO, fold-aware
    train_level3.py       # PPO Discrete(5), fold-aware; --no-l1 flag
    train_level4.py       # PPO Discrete(9), curriculum, fold-aware; --no-l1, --no-turnover, --no-vix
    train_single_level.py # Flat PPO Discrete(9), no hierarchy (NEW)
  evaluation/
    backtester.py         # 2-level and 3-level backtest
    walk_forward.py       # Per-fold retraining + evaluation
    visualizer.py         # Fold curves, heatmaps
  models/
    lstm_encoder.py       # LSTM feature extractor
    transformer_encoder.py # Transformer encoder
  data_pipeline/
    downloader.py         # yfinance + FRED download
    feature_engineering.py # 23 features (20 base + 3 macro), exports N_FEATURES
scripts/
  generate_report.py      # HTML report generator
tests/
  test_env.py
  test_agents.py
  test_integration.py
  test_pipeline.py        # Fixed for N_FEATURES=23

Ablation Flags

Macro Features (FRED)

3 columns added to FEATURE_COLUMNS (N_FEATURES=23):

• dgs10: 10-Year Treasury Constant Maturity Rate
• dff: Effective Federal Funds Rate
• baa_aaa_spread: BAA minus AAA corporate bond yield spread

Requires FRED_API_KEY env var (free from https://fred.stlouisfed.org/docs/api/api_key.html). Without it, macro columns default to 0.0.

Commands

# Train individual levels
PYTHONPATH=. python -m src.training.train_level1
PYTHONPATH=. python -m src.training.train_level2
PYTHONPATH=. python -m src.training.train_level3
PYTHONPATH=. python -m src.training.train_level3 --no-l1              # ablation
PYTHONPATH=. python -m src.training.train_level4
PYTHONPATH=. python -m src.training.train_level4 --no-l1               # ablation
PYTHONPATH=. python -m src.training.train_level4 --no-turnover          # ablation
PYTHONPATH=. python -m src.training.train_level4 --no-vix               # ablation
PYTHONPATH=. python -m src.training.train_single_level                  # flat PPO
PYTHONPATH=. python -m src.training.train_level4 --augmentation          # with data augmentation
PYTHONPATH=. python -m src.training.train_level2 --augmentation          # with data augmentation
PYTHONPATH=. python -m src.training.train_level3 --augmentation          # with data augmentation

# Walk-forward (both architectures)
PYTHONPATH=. python -m src.evaluation.walk_forward \
    --architecture both --seeds 1 --skip-rl \
    --timesteps-l2 500000 --timesteps-l3 1000000 --timesteps-l4 1000000

# Run tests
python -m pytest tests/

Reference

See docs/2026-06-01/EXECUTION_PLAN.md for Phase A/B/C execution plan with detailed file paths, line numbers, time estimates, and dependencies.