7 Forecasting and out-of-sample testing
Out-of-sample testing can be used to see if the real time investor could benefit from the in-sample results. Moreover, some researchers201 have said that out-of-sample testing can protect against data mining and overfitting, whereas others202 have stated that this is not the case, since data mining can be done for both in- and out-of-sample testing.
The procedure for out-of-sample203 testing in the current thesis will be founded in the
procedure used by Rapach, Wohar and Rangvid204. The procedure will be explained using the 4 quarter return and export regression for Denmark as an example.
observed 2Q 2000 stock return. This process is continued for the entire out-of-sample period.
The forecast error for the random walk is also calculated for the out-of-sample period.
When the forecast errors for both the model and the random walk is obtained, the Mean Squared Error, MSEM, for the estimated model is calculated using the following formula206
( ) ∑
−( )
= +
− −
−
=
K T
R t
M
Et
K R T
MSE 1 1 2, where T is the number of observations in the total data sample, both in- and out-of-sample, R is the number of observations in-sample, K as before is the estimation horizon, here 4 and t is the time. This is the mean of the squared forecast errors, since
(
T −R−K)
gives the number of out-of-sample observations. The MSERW for the random walk is calculated in the same way. When evaluating the estimated model out-of-sample against the random walk, the MSE’s are compared. The model with the lowest MSE has performed best in forecasting the stock return over the out-of-sample period.McCracken207 suggests a test value, which is founded in the MSE and can be tested using a distribution derived from a bootstrapping procedure. The current thesis excludes the use of bootstrapping, and the test will therefore not be performed. However, the size and sign of the test value holds information about the relative performance of the estimated model and the alternative model. The test value is give by208
( )
RW M MMSE MSE K MSE
R T F
MSE −
⋅
−
−
=
− . If
MSE-F is positive, this means that the estimated model has performed better than the alternative model during the out-of-sample period. However, if MSE-F is negative, the alternative model has performed best during the period. The size of the test value determines how must one model has outperformed the other.
206 Rapach et al, 2005
207 McCracken, 2004, seen in Rapach et al., 2005
208 Rapach et al., 2005
Table 7.1
Horizon K quarters 4 12 20 4 12 20 4 12 20
MSE Estimated model 0,0645 0,0928 0,0801 0,0653 0,1052 0,0896 0,0646 0,1103 0,1134 MSE Random walk 0,0269 0,0260 0,0281 0,0269 0,0260 0,0281 0,0269 0,0260 0,0281 MSE-F -0,0139 -0,0171 -0,0155 -0,0140 -0,0179 -0,0163 -0,0139 -0,0182 -0,0179 MSE Estimated model 0,0839 0,1761 0,1857 0,0840 0,1791 0,1822 0,0847 0,1833 0,1877 MSE Random walk 0,0282 0,0281 0,0345 0,0282 0,0281 0,0345 0,0282 0,0281 0,0345 MSE-F -0,0158 -0,0200 -0,0194 -0,0158 -0,0201 -0,0193 -0,0159 -0,0202 -0,0194 MSE Estimated model 0,0653 0,1417 0,1248 0,0660 0,1354 0,1190 0,0675 0,1503 0,1324 MSE Random walk 0,0246 0,0289 0,0296 0,0246 0,0289 0,0296 0,0246 0,0289 0,0296 MSE-F -0,0148 -0,0189 -0,0182 -0,0149 -0,0187 -0,0179 -0,0151 -0,0192 -0,0185 MSE Estimated model 0,0321 0,0602 0,0631 0,0372 0,0893 0,0912 0,0453 0,1466 0,1612 MSE Random walk 0,0126 0,0132 0,0152 0,0126 0,0132 0,0152 0,0126 0,0132 0,0152 MSE-F -0,0144 -0,0186 -0,0181 -0,0157 -0,0203 -0,0198 -0,0172 -0,0217 -0,0216 United Kingdom
Denmark
Out-of-sample testing
The Netherlands
France
Output - RESpy Export - px Import - pz
Table 7.1 shows the MSE for the estimated models and the random walk, and the test statistic MSE-F for all the countries. It is clear, that the forecasts for all regressions are out-performed by the random walk in predicting the stock return during the out-of-sample period, since all MSEMs are higher than the MSERWs, which give rise to the negative Fs. From the MSE-F, it can be seen that the output regressions generally perform best and the import regressions perform worst for Denmark.
Goyal and Welch209 suggest, that the performance of the out-of-sample testing can be shown graphically by plotting the cumulative difference in the squared errors of the model versus the alternative, i.e. the following
( ) (
1)
22 1
M t RW
t E
E+ − + is plotted against time. When the estimated model outperforms the random walk, the line will be upward sloping, and when the random walk outperforms the estimated model, the line will be downward sloping. This gives a very clear view of the relative performance of the two models over the out-of-sample period.
Figure 7.1.1 shows the predictability of the 4 quarter return, and from this it is clear, that during most quarters the random walk outperforms the estimated model. However, the estimated model does outperform the random walk during short periods, such as from the late 2000 to the middle of 2001, and in 2003 and 2009.
209 Goyal and Welch, 2004
Figure 7.1.1
Cumulative squared errors for 4 quarter - Denmark
-2,0 -1,8 -1,6 -1,4 -1,2 -1,0 -0,8 -0,6 -0,4 -0,2 0,0
Q1 2000 Q4 2000 Q3 2001 Q2 2002 Q1 2003 Q4 2003 Q3 2004 Q2 2005 Q1 2006 Q4 2006 Q3 2007 Q2 2008 Q1 2009 Q4 2009
Time
Cumulative squared errors
Output Export Import
Figure 7.1.2 shows the predictability over 12 quarters, and for this prediction period, the two models perform relatively equal between the beginning of 2001 and the end of 2005, and again between the beginning of 2008 to the end of the sample period. However, the random walk greatly outperforms the estimated models in 2006 and 2007, and the estimated models never really forecast stock returns better than the random walk. Additionally, it is clear to see that the output regression performs better than the other regressions.
Figure 7.1.2
Cumulative squared errors for 12 quarter - Denmark
-6,0 -5,0 -4,0 -3,0 -2,0 -1,0 0,0
Q1 2000 Q4 2000 Q3 2001 Q2 2002 Q1 2003 Q4 2003 Q3 2004 Q2 2005 Q1 2006 Q4 2006 Q3 2007 Q2 2008 Q1 2009 Q4 2009
Time
Cumulative squared errors
Output Export Import
Figure 7.1.3 gives the predictability over 20 quarters. For this prediction period, the estimated models are outperformed in the beginning of the period and during 2006 and 2007, which is the same pattern as for the 12 quarter return. The import model seems to perform relatively more poorly compared with the other models.
Figure 7.1.3
Cumulative squared errors for 20 quarter - Denmark
-6,0 -5,0 -4,0 -3,0 -2,0 -1,0 0,0
Q1 2000 Q4 2000 Q3 2001 Q2 2002 Q1 2003 Q4 2003 Q3 2004 Q2 2005 Q1 2006 Q4 2006 Q3 2007 Q2 2008 Q1 2009 Q4 2009
Time
Cumulative squared errors
Output Export Import