−1000 −500 0
Figure 5.6: Observed quantiles and IQR as a function of predicted quantiles and IQR.
5.5 Performance of the Adaptive Versions of Model 1-4
Adaptive versions of the models in Chapter4are examined in this section. These will be referred to as Model A1-A4. The updating procedure is in all cases that the knots in the direction of wind speed orpow.fcdefines the bins. For Model A1 1100 points in each bin are allowed, so the estimates will be based on 11000 points for this model. It is necessary to have this number of points in each bin for the adaptive procedure to be stable. This is probably because of the large annual variation in the air density as discussed in Chapter 4 and we will see that the model performance is still very poor. The other models allow between 800 and 1300 points in each bin. This means that the estimates are based in approximately 5000 point for all other models.
5.5.1 Reliability
Table5.4 and Figure5.7deal with reliability Model A1-A4. Both the plot and the table clearly show very large improvements in reliability. The strength of the adaptive procedure is again stressed, but the reliability of these models is actually not better than for the Basic models analyzed in the previous section.
So the reliability alone can not lead to choosing one of the more advanced
Local reliability measure
Model A1 A2 A3 4
Below 75% (test) 77.8% 77.7% 76.5% 75.7%
Below 25% (test) 27.7% 27.5 25.5% 26.2%
d(q(pow.fc,0.25)) 0.045 0.032 0.092 0.101 d(q(pow.fc,0.5)) 0.026 0.033 0.064 0.081 d(q(pow.fc,0.75)) 0.045 0.042 0.050 0.040
dqtotal(pow.fc) 0.040 0.036 0.071 0.078
d(q(hor,0.25)) 0.036 0.072 0.027 0.036 d(q(hor,0.5)) 0.033 0.068 0.032 0.036 d(q(hor,0.75)) 0.039 0.042 0.035 0.032
dqtotal(hor) 0.036 0.062 0.032 0.035
d(q(time,0.25)) 0.057 0.053 0.032 0.039 d(q(time,0.5)) 50% 0.027 0.028 0.042 0.039 d(q(time,0.75)) 75% 0.063 0.056 0.049 0.042 dtotalq(time) total 0.051 0.045 0.042 0.040
Table 5.4: Overall reliability and reliability distance in the direction of pow.fc, horizon and time for the adaptive models.
models. Table5.4 suggest that we should choose Model A3 or A4 if we should choose one of these adaptive models.
5.5.2 Skill Score and Crossings
Table5.5 shows the skill score and the number of crossings. From a skill score perspective we should choose Model A3 for the 75% quantile and Model A2 for the 25% quantile. If the interval score is considered we should choose Model A2. Model A3 and A4 have a better skill score for the 75% quantile than the Basic models from the previous section. This supports the conclusion that the Basic models seemed too simple to model the 75% quantile.
Table 5.5shows that Model A1 have 378 crossings. That is far more than the other adaptive models. A more serious problem is that it produces crossings of the magnitude of 21000kW. This should be compared with the fact that absolute value of the maximum error from WPPT is 5000kW. The problem is probably, as was also discussed in Section 4.5, the very large annual variation in air density. Which results in few observations to support estimates in some areas of the data.
A point in connections with this is that this was not really punished in the
5.5 Performance of the Adaptive Versions of Model 1-4 109
Local reliability for the adaptive versions Model 1-4
0 2000 4000
Figure 5.7: Reliability as a function of pow.fc, horizon and time for adaptive versions of the models presented in Section4.5.
Skill Score and Crossing of Model A1-4
Model A1 A2 A3 A4
ρ0.75(r) 295.9 254.2 243.1 244.8
ρ0.25(r) 226.7 216.9 231.8 234.1
ρ0.75(r) +ρ0.25(r) 522.3 471.1 474.9 478.9 Crossings (test) 378 114 7 39 min(IQR) -21252.0 -307.5 -24.0 -112.6 E(IQR<0) -1139.2 -67.3 -15.1 -53.1 Table 5.5: Numbers related to IQR for Model A1-A4
Sharpness and resolution for Model A1-A4
Model A1 A2 A3 A4
E(IQR) 1013.3 966.0 967.9 926.5
sd(IQR) 1100.2 609.4 548.8 552.1
Q(IQR,0.5) 1044.8 896.9 947.4 851.9 Q(IQR,0.05) 63.7 147.2 206.8 188.0 Q(IQR,0.95) 2091.4 1993.3 1849.3 1840.0 Table 5.6: Numbers related to IQR for Model A1-A4
reliability measures. This mean that we can not let reliability stand alone as a measure. This behavior is however punished in the loss function.
From a crossing perspective we should prefer Model A3.
5.5.3 Sharpness and Resolution
Table 5.3 gives numbers related to resolution and sharpness. Note that the resolution Model A1 is very good compared to the rest of the model. This is however due to the extreme crossings so in this case the measure award a very undesirable behavior of the model. We also see that the extreme crossings of Model A1 is not punished much by sharpness.
Figure 5.8 shows sharpness and resolution for Model A2-A4. the behavior of sharpness as a function of horizon is surprising since we would expect IQR to be an increasing function of prediction horizon. We see that it drops down at the largest prediction horizon. A possible explanation for this is that what we see is actually daily variation. We have however no way of checking this with the available data.
5.5 Performance of the Adaptive Versions of Model 1-4 111 Sharpness and resolution for Model A2-A4
20 30
Figure 5.8: Observed quantiles and IQR as functions of predicted quantiles and IQR, for Model A2-A4. Model A1 is left out since it prevents us from seing the variation of the other models.
Spread / skill relationship Model A1-A4
0 2000
Figure 5.9: Observed quantiles and IQR as functions of predicted quantiles and IQR for Model A1-A4.
5.5.4 Spread / Skill Relationship
Figure 5.9 shows observed IQR as a function of forecasted IQR. This plot is constructed in a different way than the plot for the Basic models. In these plots we group in bins of forecasted IQR with a constant length. It is seen that all the models follow the perfect line quite well, except for extreme values. This can be explained with the fact that there are few observations here and that the medians therefore not really are well defined.
The plots are also difficult to interpret because there will not be equally many observations in each bin for the different models.
Mean time used per iteration and mean number of simplex steps for the adaptive model
Model B1 B2 B3 B4 A1 A2 A3 A4
E(Time) 25% 0.15 0.09 0.06 0.07 3.06 0.23 0.16 0.48 E(Time) 75% 0.08 0.06 0.06 0.05 2.79 0.24 0.17 0.46 E(n) 25% 3.15 2.93 1.44 1.78 11.77 4.51 3.87 6.45 E(n) 75% 1.36 1.75 1.24 1.17 10.41 4.63 3.95 6.86 Table 5.7: Mean time used per iteration and mean number of simplex steps (n) for the adaptive model, B1-B4 refer to the Basic adaptive models, while A1-A4 refer to the adaptive versions of Model 1-4