Evaluation module

This module is used to evaluate the predictions of MRF. The following functions are called by the statistical_evaluation() and financial_evaluation() methods of the MRF module.

Statistical Evaluation Metrics

As statistical evaluation metrics, we use the standard MAE and MSE:

\[\begin{split}\begin{align} MAE = \frac{1}{\vert T\vert} \sum_{t \in T} |\hat{y}_{t+k|t} - y_{t+k}| \\ MSE = \frac{1}{\vert T\vert} \sum_{t \in T} (\hat{y}_{t+k|t} - y_{t+k})^2 \end{align}\end{split}\]

Where \(k\) is the forecast horizon and \(\vert T\vert\) is the cardinality of the index set \(T\). In our case this index set is the specified out-of-sample observations.

Financial Evaluation Metrics

Note

You should only use financial_evaluation() if your target variable is a financial return and you can provide the underlying tradable asset.

This means the financial evaluation functions will be next to meaningless for macro forecasting.

As a method for the financial evaluation of MRF predictions, we use a trading strategy described in 1 to generate binary long/short market signals. This strategy is agnostic to the forecast horizon used. Our trading signal \(S_t\) becomes a weighted average of directional (up/down) signals obtained by our model.

\[\begin{align} S_{t}:=\frac{1}{k} \times \sum_{j=0}^{k-1}\left(\mathbb{1}\left[\hat{y}_{t+k-j \mid t-j}>0\right]-\mathbb{1}\left[\hat{y}_{t+k-j \mid t-j}<0\right]\right) \end{align}\]

Given \(r_t\) as the daily profit associated with the trading strategy and \(T_{prof}\) as the index for profit calculation, with \(T_{end}\) as the last index in \(T_{prof}\), we can calculate Annualised Return as follows:

\[\begin{align}\label{pi_t} \Upsilon_{t} = \sum_{\tau \in [t]} r_\tau \end{align}\]
\[\begin{align}\label{Annualised} ANR = \frac{252 \times \Upsilon_{T_{end}}}{\vert T_{prof} \vert} \end{align}\]

Mean return \(\bar{r}\) and Sharpe Ratio \(SR\) are then calculated as follows:

\[\begin{split}\begin{align} \bar{r} = \frac{1}{{|T_{prof}|}} \times \sum_{t \in T_{prof}} r_t \\ SR = \sqrt{252} \times \frac{\bar{r}}{St Dev(\{r_t\}_{t \in T_{prof}})} \end{align}\end{split}\]

Maximum drawdown, \(MDD\), measures the maximum observed loss from a peak to a trough in the value of a holding. Representing the value of the portfolio to be \(\Pi_t\) = \(\Upsilon_t + 1\), \(MDD\) is calculated as follows:

\[\begin{align} \label{MDD} MDD = \underset{t\in T_{prof}} \min \left\{ \frac{\Pi_t}{ \underset{\tau \in [t]} \max \left\{ \Pi_{\tau}\right\}} - 1\right\} \end{align}\]

Code

Evaluation.collect_errors(oos_pos, actual, forecasts, k=1)

Collecting forecasting errors based on MRF forecasts and observed values of the target variable.

Parameters

  • oos_pos (-) – Represents OOS period of interst for statistical evaluation. Passed in automatically when MRF.statistical_evaluation() is called.

  • actual (-) – Actual (observed) values for the target variable over the OOS period. Passed in automatically when MRF.statistical_evaluation() is called.

  • forecasts (-) – k-period forecasted values for the target variable.

  • k (-) – Forecast horizon.

Returns

Dictionary containing forecast errors corresponding to OOS period.

Return type

  • errors (dict)

Evaluation.get_MAE(error_dict, oos_pos)

Calculating Mean Absolute Error (MAE) based on collected forecasting errors.

Parameters

  • error_dict (-) – List of forecasting errors obtained via collect_errors()

  • oos_pos (-) – Time indices of OOS period

Returns

  • MAE (float)

Evaluation.get_MSE(error_dict, oos_pos)

Calculating Mean Squared Error (MSE) based on collected forecasting errors.

Parameters

  • error_dict (-) –

  • oos_pos (-) –

Returns

  • MSE (float)

Evaluation.get_annualised_return(cumulative_profit, T_profit)

Calculating Annualised Return financial return metric.

Parameters

  • cumulative_profit (-) – Series corresponding to cumulative profit values obtained from financial_evaluation() and trading_strategy() functions.

  • T_profit (-) – Time indices corresponding to profit-generating period. Note this starts k days after OOS start, since we need a previous signal to generate profit!

Returns

Yearly profit earned over OOS period

Return type

  • annualised_return (float)

Evaluation.get_max_dd_and_date(cumulative_profit)

Calculating Maximum Drawdown financial return metric.

Parameters

cumulative_profit (-) – Series corresponding to cumulative profit values obtained from financial_evaluation() and trading_strategy() functions

Returns

Maximum Drawdown metric corresponding to OOS period

Return type

  • drawdown (float)

Evaluation.get_sharpe_ratio(daily_profit)

Calculating Sharpe Ratio financial return metric.

Parameters

daily_profit (-) – Series corresponding to daily profit values obtained from financial_evaluation() and trading_strategy() functions

Returns

Sharpe Ratio corresponding to OOS period

Return type

  • sharpe_ratio (float)

Evaluation.trading_strategy(model_forecasts, stock_price, t, k=1)

Strategy for generating binary (long/short) trading signals based on MRF predictions. This strategy is agnostic to the forecast-horizon used.

Parameters

  • forecasts (-) – k-period forecasted values for the target variable.

  • stock_price (-) – Series of stock prices corresponding to target variable (returns) during same OOS period.

  • k (-) – Forecast horizon.

Returns

Backtested daily profit corresponding to implementing MRF trading signals.

Return type

  • PL_t (pd.Series)

References

1

Ruogu Yang, Parley. Lucas, Ryan. Schelpe, Camilla. (2021). Adaptive Learning on Time Series: Method and Financial Applications. arXiv preprint arXiv: 2110.11156.