Verification plots
Climate observations and seasonal forecasting databases stored at the ECOMS User Data Gateway can be remotely accessed and loaded. Take a look at the loadeR.ECOMS wiki for more details about this data access.
Here, we illustrate an example for the CFS data. The example is focused on winter surface temperature over the Iberian Peninsula.
var <- "tas"
year.ini <- 1983
year.end <- 2008
season <- c(12,1,2)
lead.month <- 1
members <- 1:15
lonlim <- c(-10,5)
latlim <- c(35,45)We load the hindcasts from the CFS and the reference observations from WFDEI. The loading for the hindcasts take a while, but the process for the observations is faster.
prd <- loadECOMS(dataset = "CFSv2_seasonal", var=var, lonLim=lonlim,
latLim=latlim, season=season, years=year.ini:year.end,
leadMonth=lead.month, members=members, time="DD", aggr.d="mean")
obs <- loadECOMS(dataset = "WFDEI", var=var, lonLim=lonlim, latLim=latlim,
season=season, years=year.ini:year.end)Tercile plots are intended to show the performance of a forecast system along a time period and it is based on the figure developed by Diez et al. (2010). This plot depicts the predicted probability of upper, middle and lower terciles. The shading of each square represents probability (darker shade=greater probability) and the dots show the corresponding observed tercile for each of the years. Skill is represented by the ROC skill score (ROCSS) for each tercile (values are shown on the right side). ROCSS is commonly used to evaluate the performance of probabilistic systems (Joliffe and Stephenson 2003). The next version of this function will also include an additional column with the forecast for the next season (without the dot representing the observed outcome).
tercilePlot(prd, obs)
## Warning in spatialMean(mm.obj): The results presented are
## the spatial mean of the input field
## Warning in spatialMean(obs): The results presented are the
## spatial mean of the input field
For rectangular spatial domains (i.e., for fields), the spatial average is first computed (with a warning) to obtain a unique series for the whole domain. This function also admits other optional arguments such as the selection of other sets of colorbar or the elimination of the trend (detrend=TRUE/FALSE). This one is also available for the other plots.
This plot can be used to allow users to “get a sense” of what a particular ROCCS value means in terms of past performance of the probabilistic forecasts.
This simple bar graph depicts the predicted likelihood of upper, middle and lower terciles for a selected target year relative to climatology (represented by a grey line). The ROCSS is represented numerically for each tercile and it is colour coded to indicate whether there is No Skill (Red), Some Skill (Grey), or Good Skill (Blue); so as provide a salient warning to users where scores are negative. This plot is based on the currently used by MeteoSwiss in their online seasonal temperature and precipitation forecasts.
year.target <- 2008
tercileBarplot(prd, obs, year.target)
## Warning in spatialMean(mm.obj): The results presented are
## the spatial mean of the input field
## Warning in spatialMean(obs): The results presented are the
## spatial mean of the input field
Depending on the area, the positive score threshold to be remarked can be different. It is possible to indicate a positive score threshold to highlight in a different colour those values of the score above a particular value. 0.5 is the default.
year.target <- 2008
tercileBarplot(prd, obs, year.target, score.threshold=0.4)This plot shows the forecast spread for discrete time periods. The background represents the climatology for the hindcast period. The shaded areas show the central tercile (dark shade) and the maximum and minimum (light shade). To avoid overinterpretation of daily peaks, the daily data has been smoothed by means of a (centered) moving average of 31 days. Therefore, at the location of the boxplots, the background shows the monthly mean forecast (the terciles and extremes being computed over members and years). The boxplots show the spread of the monthly mean forecast (for the different members) for a selected target year. The spreadPlot function is designed to plot three types of visualisations:
-
Traditional boxplot. In this plot, the boxes represent the interquartile range (middle 50% of the ensemble members) for the each month of the target year, and the whiskers represent the extent of the forecast spread (except the outliers).
spreadPlot(prd, year.target)
We could also add to the boxplot the values of the ensemble members using crosses or any other symbol (pch option). Single symbol represents the mean for each ensemble member for each time period. They are colour coded to indicate whether they fall into the upper, middle or lower tercile.
spreadPlot(prd, year.target, add.points=T, pch=21)
-
Violin plot: The boxplots can be replaced by violin plots. This option reflects the overall shape of the distribution unveiling multimodalities in the data.
spreadPlot(prd, year.target, violin=T, add.points=T, pch=21)
-
Dot plot: In this case, each symbol represents the monthly mean for each ensemble member.
spreadPlot(prd, year.target, boxplot=F, add.points=T)
The bubble plot is a map that represents the most likely tercile in colours, the probability of that tercile with the size of the bubble (optional) and the skill of the forecast system for that tercile as transparency of the bubble (optional). Currently, the skill score used is the ROCSS. Only positive scores are shown (the negative ones –the system is worse than the climatology– are not plotted). This is based on visualisations developed by Slingsby et al. (2009).
# First interpolate the observations to model grid
obs <- interpGrid(obs, new.coordinates = getGrid(prd), method = "nearest")
## Warning in interpGrid(obs, new.coordinates = getGrid(prd),
## method = "nearest"): The new longitudes are outside the data extent
## [2016-09-27 12:34:54] Calculating nearest neighbors...
## [2016-09-27 12:34:54] Performing nearest interpolation... may take a while
## [2016-09-27 12:34:54] DoneThe bubble plot can be invoked with different levels of complexity:
-
The simpliest version only shows the most likely tercile in colours.
bubblePlot(prd, obs, year.target, size.as.probability=F, score=F)
-
The probability of the most likely tercile is added as the size of the bubble.
bubblePlot(prd, obs, year.target, size.as.probability=T, score=F)
-
The skill of the forecast system for that tercile is added as transparency.
bubblePlot(prd, obs, year.target, size.as.probability=T, score=T)
-
Pie charts in stead of bubbles can be drawn indicating the predicted likelihood of each tercile.
bubblePlot(prd, obs, year.target, pie=T, score=F)
We can add to the pie the skill of the forecast system for each tercile as transparency.
bubblePlot(prd, obs, year.target, pie=T, score=T)
- Díez, E., Orfila, B., Frías, M.D., Fernández, J., Cofiño, A.S., Gutiérrez, J.M., 2011. Downscaling ECMWF seasonal precipitation forecasts in Europe using the RCA model. Tellus A 63, 757-762. doi:10.1111/j.1600-0870.2011.00523.x
- Jolliffe, I. T. and Stephenson, D. B. 2003. Forecast Verification: A Practitioner's Guide in Atmospheric Science, Wiley, NY.
- Slingsby A., Lowe R., Dykes J., Stephenson D. B., Wood J., Jupp T. E. 2009. A pilot study for the collaborative development of new ways of visualising seasonal climate forecasts. Proc. 17th Annu. Conf. of GIS Research UK, Durham, UK, 1-3 April 2009.
