12 Uncertainty

EwE has extensive ways of treating uncertainty – but it is important to first discuss types of uncertainty.

• Background ecological variation or “noise”
  • This is variation in ecological processes that is apparently random and not strongly autocorrelated over time. In single-species dynamics and assessment, recruitment in particular typically has such variation due to unmodelled environmental factors that influence survival rate; so we may seek to predict recruitment relationships in EwE, but there will almost always be environmentally-driven variation that EwE (or any other model) does not capture.
• Parameter uncertainty
  • This is the focus for treating uncertainty in EwE – and by extension the focus of this chapter.
• Structural uncertainty
  • This can for instance be model bias: how well does the model describe the system? Typically, this can be addressed by using alternative models to make predictions about the specific management/policy question(s) that are being addressed. Within the EwE framework, this can be done by using a suite of model formulations, e.g., spanning from MICE-type models to complex formulations with a large number of functional groups and forcing functions.  Structural uncertainty can come both from how interactions are represented in the system and from “external” (the larger world within which the model is embedded) forcing factors that may or may not be recognized during model development. See also the discussion about alternative models in the research question chapter.
• Observation error
  • This, e.g., includes errors in sampling surveys that result in uncertainty in biomass estimates. Very few ecosystem models incorporate raw data such as for instance from trawl surveys. Most are processed through other models, e.g., single-species assessments, or converting satellite images to chlorophyll and primary productivity – as discussed in the Biomasses and units chapter. For this reason, very few EwE models deal directly with observation errors – though the effects of this uncertainty become evident in measures of uncertainty about parameter values derived by examining how much the parameter values can be changed without degrading the fit to the data (likelihood functions measure this).
• Implementation error
  • In Management Strategy Evaluation (MSE) this includes errors due for example to variation in catchability coefficients and uncontrolled variation in fishing effort. See the MSE chapter for details.

Most EwE models have focused on parameter uncertainty – but how does one assign uncertainty ranges to the many parameters in an ecosystem model? Consider a typical model with 30 functional groups and 5 fisheries exploiting 5 groups each. Such a model will have over 200 input parameters in the Ecopath base model alone.  That is too many to estimate individual parameter uncertainties for, and if done it would not be transparent. Assigning uncertainty based on parameter type, e.g., ±20% for biomasses, is not a realistic option. Instead, we have developed an alternative approach based on parameter “pedigree”. Pedigree can be defined as “a register recording a line of ancestors”. It is something that for instance a horse or dog may have, and it means we know where it came from. We use it to describe how well rooted a model is in local data, and with it, how uncertain the data are. The assumption is that local data are more reliable than regional or global data, guessed data are even less reliable, but the most uncertain are estimates that are derived from another model – notably those estimated by Ecopath mass-balance.

The EwE pedigree can be defined for the key Ecopath input parameters, that is biomass, production/biomass, consumption/biomass, diets and catches, see Figure 1. Each parameter has a set of classifications following the logic described above (the range from local data to model estimates), and a default parameter uncertainty is associated with each parameter-classification type. The default parameter uncertainties can be overwritten, but given that they are reasonable and that one will have to explain why one has changed them, most models for which users have defined pedigree for input parameters have used the default values.

There are two neat aspects to defining pedigree. By assigned pedigree index [0,1] for each parameter-classification type, we can estimate an overall pedigree [0,1] for a model, which describes how well rooted the model is in local data, and which can be compared to other ecosystem models. See Morisette (2007)^[1] for a neat analysis of how the quality of input data relates to complexity and stability in ecosystems.  The second aspect is that the pedigree table (see Figure 1) from the colour scale gradient directly gives an overview of how well rooted the data that are used for a model are in local data. This is a great tool for communicating what data is available from a given ecosystem. It also makes it quite clear when models feed models, i.e. when a model relies heavily on data from other models. That is generally to be avoided when possible.

Figure 1. Input screen for defining pedigree (Ecopath > Input > Tools > Pedigree) for Anchovy Bay. Colours (gradient in grey scale) in the assignment table indicates classifications, such as shown in the Classification table, which is for the production/biomass ratio. There are similar classification tables for the other input parameters, all with defined default uncertainty associated. 

Earlier versions of EwE (5) had an EcoRanger routine, which varied each of the basic Ecopath input parameters and evaluated what impact such changes had on output parameters. We have not ported that routine to later version (6+) for the simple reason that the output wasn’t interesting or credible. EcoRanger produced pages and pages of tables, but didn’t answer any questions. Questions about uncertainty have to be related to research and policy questions (see Your research question? chapter) for uncertainty analysis to be interesting and worthwhile. Beth Fulton has illustrated this by estimating that to evaluate parameter uncertainty for all input parameters for an Atlantis model^[2] would take the biggest super-computer on Earth longer than the age of the Universe. And it wouldn’t be interesting! Uncertainty has to be focused on the research/policy questions that the given model is built to address.

With this in mind, it should be clear that there are no direct uncertainty analyses in the Ecopath section of EwE. That is not because there is no uncertainty associated with Ecopath parameters (as discussed above), but because Ecopath serves as a base model for other analyses, especially Ecosim. The intention with the Ecopath model is to provide one possible parameter realization for a given ecosystem. Once we have such a one, we can throw uncertainty at it by sampling, including by generating not just one, but many Ecopath and Ecosim models, thereby providing the foundation for addressing questions about uncertainty.

Sampling of parameter values can be based on Monte Carlo (MC) techniques, which is used to estimate properties of a distribution from random sampling. As an example, a MC approach can be used to draw a large number of random samples from an unknown distribution, and calculate the sample mean of those. A major benefit of MC is thus that calculating parameters for a random sample is often much easier than calculating the parameters directly from the distribution (if at all known, that is).  Expanding on this, Markov Chain MC (MCMC) is a Bayesian approach that uses random samples to generate new random samples (hence “chain”) within a probability space. Each new sample depends only on the previous sample, with an incremental change (that’s the “Markov property”), and the more such samples, the more the final samples will resemble the original (perhaps unknown) distribution. 
We refer (without specifics) to the statistics literature for details about MC and MCMC, both of which procedures are used extensively in EwE.

Ecosim includes a MC routine (Ecosim > Tools > Monte Carlo simulation), which can be used to run Ecosim in an attempt to find a model that improves fit to time series with given parameter uncertainty for biomasses, production/biomass ratios, consumption/biomass ratios, ecotrophic efficiencies, diets, biomass accumulation or biomass accumulation rates, landings and discards.  Parameter uncertainty can be defined for each group-parameter combination, and is often populated from pedigree. The routine will draw an Ecopath input parameter combination, evaluate the derived model based on mass-balance criteria, and if the model passes these criteria, the MC routine will run Ecosim and evaluate the fit to time series data. If a model produces a better fit to the time series data than previously obtained, the model will be retained. At the end of the run, the best fit model parameter combination can then be used as an Ecopath base model, if so desired. The output from the model runs can also be used to evaluate the overall trajectories for Ecosim runs with the given parameter uncertainty.

Where the MC routine as described above evaluates one possible Ecopath model realization at the time, the CEFAS MSE plug-in for EwE (see Management strategy evaluation chapter), provides a method for developing a suite of possible Ecopath models, which subsequently can be used for resampling in comparison of policy options.

The possibility of creating multiple Ecopath base models has also been included in the EwE Ecosampler routine, which is a very versatile approach for addressing uncertainty for any routine in Ecosim and Ecospace. Ecosampler records alternative mass-balanced Ecopath models from MC, and replays these “samples” through EwE main modules and plug-ins, including Ecopath, Ecosim, Ecotracer, EcoIndicators, value chain and others). The routine captures output variation due to base input parameter uncertainty. We here refer to the Ecosampler chapter in the EwE User Guide for details.

Uncertainty in time series (e.g., environmental forcing functions, fishing effort, or biomass series) can be addressed with the Multi-sim plug-in described in the EwE User Guide, to which we refer to details. There is a tutorial in (the web and online pdf versions of) this book, that can used to get experience with the approach (see the Uncertainty in time series data tutorial).

When it comes to Ecospace, there are only few routines and examples of explicitly evaluating uncertainty. One notable example, however, was for the Roberts Bank Terminal 2 expansion project (see the Environmental impact assessment chapter of the online version of this textbook for details) where the research question related to uncertainty was how model structure and parameterization impact predictions about ecological impact of a proposed container terminal. For this analysis we ran a complex ecosystem model 5,000 times several times over as part of very comprehensive analyses of model prediction uncertainty. The Ecospace MC approach was implemented as a plug-in and will be made available for a coming release of EwE.

As a closing point, note that EwE users need to beware of what we call the “bad apple” problem, i.e. the idea that just one bad apple can cause a whole barrel to go rotten. It doesn’t always happen, it’s more of an exception, but it can happen. That is, just because most parameters are well estimated does not mean that just one or two bad ones cannot cause the whole model to make hugely incorrect predictions – especially if used to address questions that the original model wasn’t designed to answer.  For example, an early model of the Georgia Strait ecosystem fit historical data very well, but had large overestimates of Q/B and P/B for Pacific hake.  This went unnoticed until a later model developer “borrowed” those two parameter values and ran a scenario where harbour seal populations were reduced through harvesting.  That policy test resulted (because of the bad hake parameters) in a substantial hake increase, which in turn caused the simulated herring (and two salmon) populations to be driven to extinction.  To make matters worse, yet another scientist then used the bad model predictions as evidence that reducing seal populations would have counter-productive results because of the value of the seals for controlling impacts of other species like hake.  The lessons from this example include (1) test your models not just to fit data, but for their specific policy predictions, (2) use pedigrees to check for possible bad parameter estimates, and (3) do not assume that effects of multiple parameters are additive so that a few errors will not have major impact on model performance.