32 Statistical approaches for estimating vulnerability multipliers

It is becoming more common for Ecosim vulnerability multipliers and primary production anomalies to be estimated using statistical routines, with Heymans et al.^[1] demonstrating that it is best practice to estimate vulnerability multipliers by fitting model simulations to time series reference data. Longer time series are preferable as they provide an opportunity to explore important drivers of change and tend to have strong contrast in the data, which improves the model’s ability to estimate parameters, leading to more confidence in our assessment of ecosystem dynamics.

Model fitting that includes estimation of primary production anomalies is basically the ecosystem equivalent of estimating recruitment and mortality anomalies in state-space approaches to parameter estimation for single-species models, a key difference being that the anomaly estimates may be informed by correlated variation in time series patterns of multiple species.

The quality (precision, informative contrast over time) of time series data is important, especially if the fitted model is to be used for management purposes, as vulnerability multipliers (and thus predation rate changes), will be used in forward simulations to times beyond the data. In Ecosim, users can weight time series data to represent how reliable or variable time series are compared to the other reference time series. Low weights imply that the data either has high variance or is unreliable (e.g., underestimated or uncertain catches). Weightings impact the contribution of time series to the assessment of model performance, where a weight of 0 indicates that the time series will not be used in the calculation of “goodness of fit”. Weightings can be assigned based on a qualitative assessment of data pedigree (e.g., based on data origin), or by using more quantitative information, such as confidence intervals from survey estimates, the retrospective analyses of stock assessment models, or signal to noise ratio assessments^[2].

The procedure for estimating vulnerability multipliers and production anomalies that improve the fit of model simulation to calibration data is based on minimization of a sum of squares (SS) of prediction errors, which is then checked for overparameterization using the Akaike Information Index (AIC)^{[3] [4]}.

The SS is used to calculate a log likelihood criterion (Figure 1), assuming normally distributed deviations of log model predictions from log observations, evaluated at the conditional maximum likelihood estimate of the prediction error variance and scaled in the case of relative observations (y) by the maximum likelihood estimate of the relative simulation scaling factor (q) in the equation y = q x X,  where X is the absolute observation. Model fitting then proceeds by numerical search procedures to seek parameter values that minimize SS.

When generating a set of model fits under different fitting hypotheses or methods for choosing what parameters to include in the SS, AIC is then used to identify the model of best fit. AIC is a tool for model selection that penalizes for fitting too many parameters relative to the time series available for estimating the SS and is calculated as

[latex]AIC = n \cdot \ln (\frac{minSS}{n}) + 2K\tag{1}[/latex]

where n is the total number of observations, or time series values, from the loaded calibration time series and K is the number of parameters estimated. When sample size is small, there is a large probability that AIC will select models with too many estimated parameters (i.e., overfit models). The modified AIC_c can be used to address this potential overfitting by including a correction for small sample sizes

[latex]AIC_c = AIC + 2K \cdot (\frac{K-1}{n-K-1})\tag{2}[/latex]

As a rule of thumb^[5], AIC_c should be used unless n/K > ~40. In other words, unless the number of estimated parameters equates to a minimum 2-3% of the amount of data, use AIC_c. AIC_c should therefore be used when assessing EwE model performance.

A word of caution, the AIC calculations assume that the observations are independent whereas timeseries data such as typically used for ecosystem modelling have high autocorrelation. For this reason, it is advisable to test the impact on assumptions about n on model selection.

Multiple approaches have been developed to statistically estimate vulnerability multipliers (see Figure 2 in Vulnerability and vulnerability multipliers chapter). They can be estimated for predators, providing a single multiplier limit to all of a given predator’s base predation rates, and they can be estimated for individual predator-prey relationships, which assumes that the multiplier limits are heterogeneous across prey. This choice tends to be associated with user preference, ecological justification, or determined based on the approach that produces the best fit model. Whether estimating predator or predator-prey vulnerability multipliers, there are a few ways to select which vulnerability multipliers should be estimated:

1. manual selection based on a priori knowledge or species priority;
2. select vulnerability multipliers for groups with calibration time series; or
3. select the most sensitive vulnerability multipliers (i.e., those that when changed have the largest impact on SS).

Manually selecting vulnerability multipliers allows for an early integration of ecological information but may lead to a sub-optimal model fit if the SS is not sensitive to the selected multipliers. Conversely, the sensitivity search may optimise model fit but it is purely statistical and does not know what makes sense ecologically. Only estimating vulnerability multipliers for groups with time series acknowledges that, to some degree, the parameter should be constrained by the available time series. The level of group connectedness within the food web (e.g., group consumption and predation) may also constrain the parameter search if changes in vulnerability multipliers impact the  contribution of other groups.  Groups that do not have informative time series or, have low connectedness in the food web, have widely variable estimated vulnerability multipliers – search routines can change those without any penalty incurred.

Choosing how many vulnerability multipliers to estimate, without overfitting is another point of confusion and discussion. The number of vulnerability multipliers that can be potentially estimated is often significantly more than the data available to constrain simulations. EwE best practices suggest that a conservative number of Degrees of Freedom (DoF) and therefore parameters to estimate is one less than the number of calibration time series available^[7]. This approach recognizes that values within time series are highly autocorrelated, viewing each time series as an “independent observation”, but  the  approach could be overly conservative, especially if long time series are available– especially if the contrast (ups and downs) and are not just one-way trajectories.

Both manual and automated statistical calibration routines are available in Ecosim to search for vulnerability multipliers. The manual approach can be arduous when testing multiple fitting hypotheses (e.g., with or without fishing effort or primary production anomalies) as the number of plausible fitting combinations can easily reach the hundreds, if not thousands, increasing the likelihood of user error. In the past, users have overcome this issue by only testing the nth fitting scenario (e.g., 5, 10, 15 vulnerability multipliers etc.)^[8] However this approach risks overlooking the vulnerability multiplier combination, which produces the best statistical fit. The stepwise fitting procedure developed by Scott et al.^[9] automates this process, allowing for a broad exploration of the parameter space which accelerates the process and removes the problem of user error. Recent improvements to the automated approach have increased the computational speed by enabling multiple fitting scenarios to be tested simultaneously using computers multithreading capabilities (J. Steenbeek, pers. comm.).

Novel approaches to estimate vulnerability multipliers using the manual and automated fitting routines have also been developed for two EwE models which are being used operationally to inform fisheries catch advice. Bentley et al.,^[10] employed an approach, which combined searches for predator vulnerability multipliers and predator-prey vulnerability multipliers, whereas the approach developed by Chagaris et al.^[11] uses the manual fitting tool in Ecosim to iteratively estimate the  most sensitive predator-prey vulnerability multipliers over multiple sequential (repeated) tuning iterations.^[12].

It is worth reiterating that statistical estimation of vulnerability multipliers does not necessarily have any bearing on ecology. While it is possible to exclude vulnerability multipliers from the search routine, there is currently no mechanism to include prior information or ecologically sensible bounds to constrain the limits for vulnerability multipliers included in the search routine. A judgement evaluation following the formal estimation of vulnerability multipliers should be applied to:

1. reflect on the ecological assumptions attached to estimated vulnerability multipliers,
2. assess how realistic functional group simulations are (in hindcast and future), and
3. understand and fix issues with model structure and parameterization (Figure 1) .

It is possible to view the fit of each functional group to calibration time series and its contribution to the overall SS in Ecosim via the Ecosim > Output > Ecosim group plots form. This is often used to screen issues with model simulations, such as contradicting trends or misalignment in initial time steps, and direct fixes.

However, what is often not accounted for when estimating vulnerability multipliers is their impacts on the advice products such as F_MSY or food web indicators. The focus is often only on the goodness of fit of the model, but the impacts of estimated vulnerability multipliers on predictions and reference points should be evaluated^[13]. We next provide two case studies to explore how alternate fitting approaches impact the emergence of vulnerability multipliers and how vulnerability multipliers impact model outputs.