OMSCS MACHINE LEARNING PROJECT

Algorithmic Trading:
Validation to Reality

Building a Random Forest Trading Agent that learned to navigate the 2008 Financial Crisis. A journey from beating the benchmark to uncovering the fragility of "perfect" models.

The Hypothesis

Can extreme regularization (Leaf Size > 100) force a model to learn macro-regimes instead of daily noise, thus preventing overfitting?

The Constraint

Compliance testing. The model must pass the "Sine Fast Noise" unit test, simulating high-volatility environments where fragile models break.

Manual (IS)
+4.42%
Baseline Strategy
Learner (IS)
+17.1%
In-Sample Training
Learner (OOS)
+3.19%
Out-of-Sample Test
Benchmark
-8.36%
Market Performance

01 The "Miracle" Experiment

R&D Hypothesis

Seeking Perfect Generalization

I hypothesized that extreme regularization would force the model to ignore daily noise. By setting Leaf Size = 140 (very large), the model learned macro-regimes rather than micro-patterns.

The result was unprecedented: The Out-of-Sample return (+13.61%) actually exceeded In-Sample performance (+12.77%). The model was not fragile; it was exceptionally robust.

OOS Return

+13.61%

Status

Abandoned

Failed Unit Test Constraints

02 Production Conflict

Performance vs. Compliance

Why abandon a model with +13.6% OOS return?

The Trap: To achieve those returns, the model utilized a low threshold (0.011). While profitable on market data, this configuration failed the strict SINE_FAST_NOISE unit test required for deployment.

The grading script (simulating a high-stress environment) demanded a safety margin that the "Miracle" model couldn't provide without breaking. I was forced to pivot.

logic.py
                                def get_threshold(self):
                                    # ❌ "Miracle" Settings (Failed Unit Test)
                                    # threshold = 0.011 (Too low for noisy env)

                                    # ✅ Production Settings (Passed Test)
                                    # Force higher threshold for safety
                                    
                                    threshold = 0.011 + (self.impact * 2)
                                    return threshold

03 The Reality of Production

Final Strategy

Compromised on Leaf Size 38.

Although it has higher variance and lower returns than the "Miracle" model, it meets all operational robustness checks (Sine Fast Noise).

Defense Mechanism

The "Flat Line" in 2010 proves the model correctly identified low-confidence regimes and stayed in Cash.