T2-Econ

Imported from ArjunDivecha/T2-Econ (AGENTS.md). Install upstream with npx skills add ArjunDivecha/T2-Econ. Copyright stays with the author.

AGENTS.md

This file provides guidance to Codex (Codex.ai/code) when working with code in this repository.

Commands

Running Individual Steps — Classic T2

python "Step Zero Create P2P Scores.py"
python "Step One Create T2Master.py"
python "Step Two Create Normalized Tidy.py"
python "Step Two Point Five Create Benchmark Rets.py"
python "Step Three Top20 Portfolios.py"
python "Step Four Create Monthly Top20 Returns.py"
python "Step Five 60 Month Optimal Portfolios.py"
python "Step Seven Visualize Factor Weights.py"
python "Step Eight Write Country Weights.py"
python "Step Nine Calculate Portfolio Returns.py"
python "Step Ten Create Final Report.py"
python "Step Fourteen Target Optimization.py"
python "Step Fifteen Market Regime Analysis.py"

Running Individual Steps — Pure Econ

python "Step Zero Build Econ.py"          # build Econ.xlsx from ASADO DuckDB
python "Step Two Econ Create Tidy.py"
python "Step Three Econ Top20 Portfolios Fast.py"
python "Step Four Econ Create Monthly Top20 Returns FAST.py"
python "Step Five Econ FAST.py"
python "Step Six Econ Create Country Alphas from Factor alphas.py"
python "Step Seven Econ Visualize Factor Weights.py"
python "Step Eight Econ Write Country Weights.py"
python "Step Nine Econ Calculate Portfolio Returns.py"
python "Step Ten Econ Create Final Report.py"
python "Step Fourteen Econ Target Optimization.py"

Archived Steps

The following steps have been moved to Archive/ and are not part of the main flow:

• Step Six T2 Factor Timing Top3.py
• Step Eleven Compare Strategies.py
• Step Twelve MultiPeriod Forecast.py
• Step Thirteen Create Country alphas from Factor Alphas.py
• Step Run All.py

Testing and Development

# Install dependencies
pip install -r requirements.txt

# Run Jupyter notebooks for interactive analysis
jupyter lab

Architecture

Pipeline Overview

The T2 Factor Timing system is a sequential pipeline for momentum-based country selection and portfolio optimization:

1. Data Preparation (Steps 0-2.5):

   • Ingests Bloomberg financial data and P2P scores
   • Creates normalized data with quality enhancements (outlier detection, winsorization)
   • Generates benchmark returns for performance comparison

2. Portfolio Construction (Steps 3-5):

   • Identifies top 20 countries by momentum across multiple lookback periods
   • Calculates monthly returns for each momentum portfolio
   • Performs 60-month rolling window optimization to find optimal factor weights

3. Portfolio Implementation (Steps 7-9):

   • Visualizes factor weights over time
   • Translates factor weights to country-level allocations
   • Calculates final portfolio returns with rebalancing

4. Reporting and Optimization (Steps 10, 14-15):

   • Creates comprehensive performance reports with visualizations
   • Optimizes country weights using CVXPY with turnover constraints
   • Analyzes strategy performance across market regime conditions

Key Data Flow

Bloomberg Data → T2 Master → Normalized Data → Top20 Portfolios → Factor Returns → 
Factor Weights → Country Weights → Portfolio Returns → Performance Reports → 
Target Optimization → Market Regime Analysis

Critical Implementation Details

Data Quality Pipeline: Step One implements sophisticated data cleaning with forward-filling, local outlier detection (±3σ rolling windows), and global winsorization. Market cap data receives special handling to preserve large-cap influence.

Portfolio Optimization: The strategy uses rolling window optimization to find optimal factor weights that maximize risk-adjusted returns. Factor weights are constrained by maximum allocation limits defined in Step Factor Categories.xlsx.

Target Optimization: Step Fourteen implements CVXPY-based optimization that balances three objectives: maximizing portfolio alpha, minimizing drift from rolled-forward weights, and minimizing transaction costs from turnover.

Market Regime Analysis: Step Fifteen analyzes strategy performance across different market conditions (Bull/Bear, High/Low Volatility, Economic Expansion/Contraction) and identifies which factors drive performance in each regime.

File Dependencies

• Input data must be in Excel format with specific sheet names
• All intermediate outputs are Excel files for compatibility
• Visualizations saved as PDFs in outputs/visualizations/
• Date columns must be datetime-indexed throughout the pipeline

Development and Debugging Tools

Archived Analysis Files (in Archive/ directory):

• Step Six T2 Factor Timing Top3.py - Factor timing with adaptive rotation
• Step Six Grid Search.ipynb - Jupyter notebook for parameter optimization
• Step Run All.py - Sequential pipeline execution script
• Various experimental and comparison analysis files

Logging: All scripts generate detailed logs to console and T2_processing.log

Data Quality and Error Handling

• Missing country data is filled with the mean of available countries
• Forward-filling handles temporal gaps in data
• Outlier detection uses ±3σ rolling windows with winsorization
• Market cap data receives special handling to preserve large-cap influence
• All processing steps include comprehensive error handling and validation

Learned User Preferences

• Align Econ month-end dates to the same reporting convention as other T2 files (e.g. a calendar month-end such as 2/28/2026 is stored as the matching first-of-following-month style such as 3/1/2026 used elsewhere in the pipeline).
• Keep the full pipeline restricted to the external-signal-aligned sample window once dates are aligned (do not silently extend analysis beyond tradable return coverage).
• For the Econ factor-category workbook, include all listed factors, including z-scored variants.
• Sign-flip “risk” category variables so their direction matches the intended economic interpretation alongside other factor groups.
• Classic T2 Step Three Top20 Portfolios Fast.py regression support (step_three_regression_utils.py) uses simple monthly cross-sectional regression (one OLS per factor per month), not Fama-MacBeth; regress country excess returns (1M return minus the equal-weight benchmark) on factor scores, and treat the monthly slope series as the factor return in charts and summaries—do not subtract the benchmark again from those slopes.
• LASSO / ElasticNet overlay was removed from Step Three (2026-04-06); only sort-based and univariate OLS regression remain.

Learned Workspace Facts

• The active Econ track uses Econ names for scripts, outputs, helper modules, and docs. Pure-signal factors come from Econ.xlsx (and derived tidy outputs); country returns and other return fields still come from T2 Master / existing market inputs, not from Econ.
• Step Zero Build Econ.py generates Econ.xlsx directly from the ASADO DuckDB warehouse (/Users/arjundivecha/Dropbox/AAA Backup/A Working/ASADO/Data/asado.duckdb). It reads from explicit raw factor tables — external_factors, extended_factors, imf_factors, macrostructure_factors, plus the 9 MS_* ETF / passive-flow Bloomberg variables — and never from views (feature_panel, unified_panel, normalized_panel) or from optimizer-output tables (factor_returns, factor_top20_membership, country_factor_attribution). This is a structural guarantee against the optimizer-output → optimizer-input cycle; do not "simplify" by switching the read to feature_panel even if the row count looks identical, because ASADO's collect_optimizer_returns.py ingests this pipeline's outputs back into DuckDB. Atomic write with timestamped backup of the previous Econ.xlsx to ./backups/. Replaces the old manual Country_Data_Book.xlsx → Econ.xlsx copy/rename.
• Step Zero Build Econ.py dependencies: duckdb, pandas, openpyxl. The miniforge base env at /opt/homebrew/Caskroom/miniforge/base/bin/python had pandas and openpyxl already; duckdb was added (2026-04-29). If a future env is missing it, install with pip install duckdb rather than rerouting to a different interpreter.
• Step Two Econ Create Tidy.py lags Econ signal-panel dates by one month before merging to T2 forward returns; do not remove this lag unless source-vintage semantics prove the sheets are already next-month-labelled.
• Step Two Econ Create Tidy.py also sign-flips an explicit ECON_INVERT_SHEETS set for lower-is-better variables before CS/TS normalization, so higher normalized scores mean more attractive countries.
• Step Two Econ Create Tidy.py emits *_D12_CS and *_D12_TS variants for the explicit ECON_CHANGE_12M_SHEETS set; D12 is a calendar 12-month change computed after any inversion, so positive means improving for lower-is-better variables.
• Econ factors are maintained as cross-sectional (CS) and time-series (TS) variants per variable.
• The user-aligned analysis start for Econ-wide work is 2015-09-01 (end date follows Econ coverage after date harmonization).
• Step Factor Categories Econ.xlsx is the Econ counterpart to Step Factor Categories.xlsx for caps, groupings, and factor lists.
• Econ.xlsx includes an INDEX metadata sheet; Step Two Econ Create Tidy.py skips documentation/index sheets and only processes wide monthly factor panels (dates in column A).
• When raw Top20 or portfolio outputs disagree with older archived implementations, treat Step Two Econ tidy/normalization as a primary place to reconcile definitions before changing Step Three.
• T2+Econ combined Top 3 branch: equal-weight the top three countries per factor per month using Step Three T2 Econ Combined Top3 Portfolios Fast.py, Step Four T2 Econ Combined Top3 Returns FAST.py, and Step Five T2 Econ Combined Top3 FAST.py, producing T2_Econ_Combined_Top3_* outputs parallel to the Top20 combined filenames.
• T2_Econ_Combined_Optimizer.xlsx (Step Four combined output) can have many more columns than true factor signals (for example ~196 columns vs ~168 factors): extras include raw cumulative return fields such as 3MRet, 6MRet, 9MRet, 12MRet and other non-factor variables. Step Five T2 Econ Combined FAST.py must restrict optimizer columns to those listed in Step Factor Categories T2 Econ Combined.xlsx; any column not in that workbook falls back to a default max weight of 1.0 and can massively inflate reported backtest returns.
• Step Five T2 Econ Combined FAST.py defines USE_COVARIANCE (default False). When it is False or LAMBDA is 0, no sample covariance matrix is built and the objective is scaled expected return minus HHI penalty only (no cp.quad_form risk term); set USE_COVARIANCE = True and LAMBDA > 0 to restore the mean–variance covariance term.
• Econ Top20.xlsx (Step Three Econ) uses sheets Full_Sample, Trailing_1Y, Trailing_3Y, Trailing_5Y; use Full_Sample where code assumed a single full-history sheet. Step Four Econ also writes Econ_RSQ.xlsx (Monthly_RSQ): R² of OLS on 12-month trailing cumulative filled net returns (aligned with the optimizer); the first 11 monthly rows have blank RSQ cells. Step Five Econ FAST.py drops all-NaN months after loading Econ_Optimizer.xlsx (e.g. stub incomplete last row) before trailing-window covariance, symmetrizes Σ before cp.quad_form, and logs dropped dates.
• For factor redundancy, pairwise correlation on Econ_Optimizer.xlsx Monthly_Net_Returns measures co-movement of backtest return series; redundancy in the signal panel should use Econ_Factors_MasterCSV.csv via mean monthly cross-sectional correlation between factors—the two views can differ materially (e.g. defensive vs risk structure in the panel vs returns).