Skip to content

Docs: runnable model tasks and verified book links #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
stefan-jansen wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Docs: runnable model tasks and verified book links #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ __pycache__/
.pytest_cache/
.ruff_cache/
.coverage
coverage.json
htmlcov/
dist/
build/
Expand Down
96 changes: 21 additions & 75 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
[![PyPI](https://img.shields.io/pypi/v/ml4t-models)](https://pypi.org/project/ml4t-models/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Finance-native model implementations for latent-factor estimation, stochastic discount factor learning, direct asset prediction, and end-to-end portfolio learning.
Finance-specific models for asset pricing, prediction, and portfolio learning.

Documentation: [ml4trading.io/docs/models](https://www.ml4trading.io/docs/models/)

Expand Down Expand Up @@ -69,96 +69,42 @@ Documentation tools are contributor dependencies. From a source checkout, run

## Quick Start

### 1. Latent-Factor Forecast Pipeline
The base package can produce a first forecast on a small synthetic panel without credentials or
an accelerator. Run this complete example with `pip install ml4t-models`:

```python
import numpy as np

from ml4t.models import (
BetaLambdaMapper,
CrossSectionBatch,
ExpandingMeanFactorForecaster,
IPCAConfig,
IPCAModel,
LatentFactorForecastPipeline,
PCAConfig,
PCAModel,
PersistentPanelBatch,
)

batch = CrossSectionBatch(
characteristics=np.random.randn(24, 200, 12),
returns=np.random.randn(24, 200),
timestamps=tuple(range(24)),
asset_ids = tuple(f"asset_{i}" for i in range(6))
train = PersistentPanelBatch(
returns=np.random.default_rng(1).normal(scale=0.02, size=(12, 6)),
timestamps=tuple(f"2024-{month:02d}" for month in range(1, 13)),
asset_ids=asset_ids,
)

future = PersistentPanelBatch(timestamps=("2025-01", "2025-02"), asset_ids=asset_ids)
pipeline = LatentFactorForecastPipeline(
model=IPCAModel(IPCAConfig(n_factors=3)),
model=PCAModel(PCAConfig(n_factors=2)),
forecaster=ExpandingMeanFactorForecaster(),
mapper=BetaLambdaMapper(),
)
pipeline.fit(batch)
prediction = pipeline.predict(batch)

print(prediction.asset_forecast.expected_returns.shape)
# (24, 200)
```

### 2. Weight-Native Stochastic Discount Factor

```python
import numpy as np

from ml4t.models import (
CrossSectionBatch,
StochasticDiscountFactorConfig,
StochasticDiscountFactorModel,
)

batch = CrossSectionBatch(
characteristics=np.random.randn(36, 300, 16),
returns=np.random.randn(36, 300),
context_features=np.random.randn(36, 8),
timestamps=tuple(range(36)),
)

model = StochasticDiscountFactorModel(
StochasticDiscountFactorConfig(checkpoint_epochs=(256, 512, 768, 1024))
)
model.fit(batch)
state = model.extract(batch, checkpoint=1280)

print(state.asset_weights.shape)
# (36, 300)
pipeline.fit(train)
forecast = pipeline.predict(future).asset_forecast.expected_returns
assert forecast.shape == (2, 6) and np.isfinite(forecast).all()
print(forecast.shape) # (2, 6)
```

### 3. End-to-End Portfolio Learning

```python
import numpy as np

from ml4t.models import LSTMPortfolioConfig, LSTMPortfolioModel, PortfolioSequenceBatch

batch = PortfolioSequenceBatch(
features=np.random.randn(8, 63, 20, 10),
returns=np.random.randn(8, 63, 20),
timestamps=tuple(range(63)),
asset_ids=tuple(f"asset_{i}" for i in range(20)),
)

model = LSTMPortfolioModel(LSTMPortfolioConfig(max_iters=20, checkpoint_every=5))
model.fit(batch)
weights = model.predict(batch, checkpoint=20)

print(weights.weights.shape)
# (8, 63, 20)
```

### 4. Hand Off Predictions To The Rest Of ML4T

```python
from ml4t.models import predictions_frame_from_asset_forecast, write_backtest_frames

frame = predictions_frame_from_asset_forecast(prediction.asset_forecast)
write_backtest_frames("artifacts/run_001", predictions=frame)
```
The forecast uses the training factor history and preserves the future dates and asset order.
It does not imply trading performance. The [Quickstart](docs/getting-started/quickstart.md)
explains the result, and the [Book Guide](docs/book-guide/index.md) links to pinned teaching files.
The `deep` extra is needed for neural models; the `integration` extra adds Polars and Specs support.

## Model Families

Expand Down
13 changes: 12 additions & 1 deletion docs/api/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ The package root re-exports the main model classes, configs, batches, results, a
## Stability

The [API Stability](../reference/api-stability.md) page defines the public
surface intended to remain stable through the `0.1` beta series.
surface for the `0.1` stable line.

## Integration

Expand All @@ -92,3 +92,14 @@ surface intended to remain stable through the `0.1` beta series.
| `ml4t.models.stochastic_discount_factor` | weight-native SDF estimation and return projections |
| `ml4t.models.asset_prediction` | direct asset-level predictors |
| `ml4t.models.portfolio` | end-to-end portfolio learners |

## Portfolio model signatures

The portfolio models are imported lazily from `ml4t.models`. Their class signatures are rendered
here explicitly so all three supported allocators are covered by the generated reference.

::: ml4t.models.portfolio.linear.LinearFeaturePortfolioModel

::: ml4t.models.portfolio.lstm.LSTMPortfolioModel

::: ml4t.models.portfolio.deep_portfolio.DeepPortfolioModel
156 changes: 38 additions & 118 deletions docs/book-guide/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,132 +1,52 @@
# Book Guide

`ml4t-models` is the library form of the model families developed manually in the book notebooks.
The public companion repository at revision
[`d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb`](https://github.com/stefan-jansen/machine-learning-for-trading/tree/d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb)
provides the teaching files below. Every linked path was checked against that revision's Git tree.
These notebooks explain methods and often use their own data and dependencies. None of the Chapter 14
teaching notebooks below imports `ml4t.models`; run the library's small examples for its API.

![From The Factor Zoo To A Library Taxonomy](../images/figure_14_1_factor_zoo_to_discipline.jpeg)
## Latent factors and factor forecasts

The goal is not to hide the teaching implementation. The goal is to:
| Public book file | What it does | Related library task |
|---|---|---|
| [IPCA notebook](https://github.com/stefan-jansen/machine-learning-for-trading/blob/d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb/14_latent_factors/04_ipca.ipynb) | Manually teaches characteristic-dependent betas and factor forecasts. | [Fit a latent-factor pipeline](../user-guide/latent-factor-pipelines.md) with `IPCAModel` and a separate forecaster. |
| [Risk-premium PCA notebook](https://github.com/stefan-jansen/machine-learning-for-trading/blob/d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb/14_latent_factors/05_rp_pca.ipynb) | Manually teaches pricing-aware factor extraction. | [Choose a latent-factor model](../user-guide/latent-factor-models.md) with `RPPCAModel`. |
| [Conditional autoencoder notebook](https://github.com/stefan-jansen/machine-learning-for-trading/blob/d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb/14_latent_factors/06_conditional_autoencoder.ipynb) | Manually builds a neural conditional factor model. | [Choose a latent-factor model](../user-guide/latent-factor-models.md) with `CAEModel`; the library requires `deep`. |
| [Case-study insights notebook](https://github.com/stefan-jansen/machine-learning-for-trading/blob/d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb/14_latent_factors/09_case_study_insights.ipynb) | Illustrates analysis of stored case-study results, not a first API example. | [Hand results downstream](../user-guide/integration.md). |

- show the architecture and mathematics clearly in the chapter notebooks
- use the library for repeatable case-study execution and downstream integration
The book's [case-study library bridge](https://github.com/stefan-jansen/machine-learning-for-trading/blob/d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb/case_studies/utils/latent_factors/library_bridge.py)
*does* import `ml4t.models` for PCA, IPCA, CAE, SDF, and SAE runs. It is case-study integration
code with data and registry prerequisites, not a standalone quickstart.

## Chapter Mapping
## SDF and direct prediction

### Chapter 14: Latent Factors
| Public book file | What it does | Related library task |
|---|---|---|
| [Adversarial SDF notebook](https://github.com/stefan-jansen/machine-learning-for-trading/blob/d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb/14_latent_factors/07_stochastic_discount_factor.ipynb) | Manually teaches phase-aware SDF training. | [Estimate SDF weights](../user-guide/stochastic-discount-factor.md) with `StochasticDiscountFactorModel`. |
| [Supervised autoencoder notebook](https://github.com/stefan-jansen/machine-learning-for-trading/blob/d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb/14_latent_factors/08_supervised_autoencoder.ipynb) | Manually teaches a direct supervised predictor. | [Predict asset signals](../user-guide/direct-asset-prediction.md) with `SAEModel`. |

The latent-factor chapter corresponds most directly to:
These neural notebooks need PyTorch and book data. Their full training runs are longer than the
small CPU examples in this site's task guides. The book's targets and splits may also differ from
the synthetic examples; results are not directly comparable.

- `PCAModel`
- `RPPCAModel`
- `IPCAModel`
- `CAEModel`
- `StochasticDiscountFactorModel`
- `SAEModel` as supervised autoencoder direct prediction
## Portfolio learning

The key conceptual transition from the notebooks to the library is:
| Public book file | What it does | Related library task |
|---|---|---|
| [Deep portfolio optimization](https://github.com/stefan-jansen/machine-learning-for-trading/blob/d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb/17_portfolio_construction/11_dl_portfolio_allocation.ipynb) | Illustrates a related neural allocation workflow, without calling this library. | [Learn portfolio weights](../user-guide/portfolio-learning.md). |
| [VLSTM portfolio](https://github.com/stefan-jansen/machine-learning-for-trading/blob/d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb/17_portfolio_construction/12_vlstm_portfolio.ipynb) | Manually teaches variable selection and sequence allocation. | [Learn portfolio weights](../user-guide/portfolio-learning.md) with `LSTMPortfolioModel`. |
| [DeePM regime robustness](https://github.com/stefan-jansen/machine-learning-for-trading/blob/d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb/17_portfolio_construction/13_deepm_regime_robust.ipynb) | Manually teaches a related DeePM architecture; it is not an exact library implementation. | [Learn portfolio weights](../user-guide/portfolio-learning.md) with `DeepPortfolioModel`. |

- notebook exposition may derive the math and architecture step by step
- library code enforces the clean separation between:
- structural extraction
- factor forecasting
- asset mapping
Portfolio notebooks use book-specific data and longer neural training. Start with the library's
[linear CPU example](https://github.com/ml4t/models/blob/main/examples/portfolio_learning.py)
for an observable result.

That separation matters most for `IPCAModel` and `CAEModel`. In the teaching notebooks, it
is helpful to show the full architecture and fitted-return logic step by step. In the
library, the corresponding production object is the two-step pipeline:
## Data and downstream boundaries

```text
structural estimator -> factor-premium forecaster -> asset mapper
```
The library's [data contracts](../user-guide/data-contracts.md) preserve timestamps and asset
identity. [Integration](../user-guide/integration.md) converts predictions and weights to frames for
`ml4t-diagnostic` and `ml4t-backtest`. The Chapter 14 case-study insights notebook illustrates
analysis after model runs, but it does not replace those packages' guides or APIs.

### Chapter 17: Portfolio Construction

The end-to-end allocation family corresponds to:

- `LinearFeaturePortfolioModel`
- `LSTMPortfolioModel`
- `DeepPortfolioModel`

These models are designed to connect naturally to:

- Chapter 18 cost modeling
- Chapter 19 risk controls
- Chapter 20 strategy analysis

## Why The Library Split Matters

The book often needs to compare multiple modeling ideas side by side:

- latent-factor models
- no-arbitrage SDF models
- direct signal models
- end-to-end allocation models

The library turns those into explicit families instead of treating them as one generic “deep learning model.”

## Case Studies

The case studies are intended to act as:

- integration tests
- realistic pressure tests for the API
- examples of how to hand model outputs into `ml4t-backtest` and `ml4t-diagnostic`

They should not define the public API by accident.

## Compatibility Status

The `0.1.0` stable line is validated against the Chapter 14 teaching flow and the shared case-study
latent-factor bridge.

| Book surface | Validation status |
|---|---|
| `14_latent_factors/04_ipca.ipynb` | full notebook execution passed |
| `14_latent_factors/05_rp_pca.ipynb` | full notebook execution passed |
| `14_latent_factors/06_conditional_autoencoder.ipynb` | full notebook execution passed |
| `14_latent_factors/07_stochastic_discount_factor.ipynb` | full notebook execution passed |
| `14_latent_factors/08_supervised_autoencoder.ipynb` | Papermill smoke execution passed; full production training is long-running |
| `14_latent_factors/09_case_study_insights.ipynb` | full notebook execution passed |
| `case_studies.utils.latent_factors.library_bridge` | synthetic PCA, IPCA, CAE, SAE, and SDF bridge smoke checks passed |

The teaching notebooks keep hand-built implementations where that improves exposition. The
case-study path uses `ml4t-models` through the shared latent-factor bridge so the same
contracts are exercised in walk-forward validation and registry-backed analysis.

## Case-Study Validation

The beta gate also checks that each case-study family can execute its model-specific
latent-factor notebooks through the shared bridge.

| Case study | Validation status |
|---|---|
| ETF returns | PCA, IPCA, SDF, and SAE cached executions passed; CAE passed in cached three-fold validation mode |
| US firm characteristics | IPCA, CAE, SDF, and SAE passed in cached three-fold validation mode |
| S&P 500 option analytics | PCA, IPCA, CAE, SDF, and SAE passed in cached three-fold validation mode |

The ETF CAE notebook also reached the cached full-fold execution path and loaded the
registry-backed model outputs before the notebook kernel exited while processing the large
cached result set. The three-fold validation run exercises the same library bridge,
checkpoint handling, prediction schema, and registry persistence path with a bounded
runtime footprint.

## Evaluation Boundary

Case-study IC reporting is delegated to `ml4t-diagnostic`:

- fold-level scoring calls `ml4t.diagnostic.metrics.cross_sectional_ic`
- pooled model-analysis summaries call `cross_sectional_ic` and `cross_sectional_ic_series`
- model outputs are converted into `PredictionsFrame`, `SignalsFrame`, `WeightsFrame`, and
`ml4t-backtest` handoff payloads by library adapters

`ml4t-models` remains responsible for fitting and output contracts. Statistical diagnostics
and execution simulation remain owned by `ml4t-diagnostic` and `ml4t-backtest`.

## Recommended Reading Order

If you are moving from the book notebooks to the library:

1. [Data Contracts](../user-guide/data-contracts.md)
2. [Latent-Factor Pipelines](../user-guide/latent-factor-pipelines.md)
3. [Stochastic Discount Factor](../user-guide/stochastic-discount-factor.md)
4. [Portfolio Learning](../user-guide/portfolio-learning.md)
5. [Integration](../user-guide/integration.md)
The [Quickstart](../getting-started/quickstart.md) is the first runnable library workflow.
Loading
Loading