{"id":511,"date":"2023-10-14T14:28:13","date_gmt":"2023-10-14T18:28:13","guid":{"rendered":"https:\/\/pressbooks.bccampus.ca\/eweguide\/chapter\/addressing-uncertainty\/"},"modified":"2024-05-23T16:58:17","modified_gmt":"2024-05-23T20:58:17","slug":"addressing-uncertainty","status":"publish","type":"chapter","link":"https:\/\/pressbooks.bccampus.ca\/eweguide\/chapter\/addressing-uncertainty\/","title":{"raw":"Addressing Uncertainty","rendered":"Addressing Uncertainty"},"content":{"raw":"
\r\n

General Practices<\/h1>\r\nIt is generally a good practice \u2013 if not mandatory \u2013 to explicitly acknowledge sources of uncertainty in data and modeling exercises, make reasonable attempts to quantify it, and caveat the model results as appropriate.\r\n\r\nAn ever-increasing body of literature categorizes the various types of uncertainty related to the development and operation of ecosystem models (Cariboni et al., 2007; Harwood and Stokes, 2003; Payne et al., 2016; Spence et al., 2016; Steenbeek et al., 2021). A useful typology is: (1) model \/ structural, (2) stochastic-, (3) data-, and (4) parameter uncertainty (Gregr and Chan, 2015). Model uncertainty occurs due to potential incomplete or inaccurate abstractions embedded in the structure of the model. Ensemble modeling approaches can be an effective approach to better understanding uncertainty due to implicit architecture of a model: multiple models are used, each representing a different method of abstracting and representing the ecosystem, e.g., Scales et al. (2016), Spence et al. (2018), Lotze et al. (2019) and Tittensor et al. (2021). Structural uncertainty can also be explored using a single model by modifying model equations, for example by aggregating\/disaggregating compartments, changing functional relationships and\/or modifying basic function process, e.g., linear fishing mortality (Coll et al., 2020; Hansen et al., 2019; Pantus, 2007). Stochastic uncertainty occurs due to chaotic or \u2018noisy\u2019 process signals, non-stationary patterns and relationships between variables (Deyle et al., 2016; Litzow et al., 2018; Walters, 1987), or uncertainty in processes represented by the model (e.g., recruitment, (Spence et al., 2016). Data uncertainty results from measurement and observation error and can be propagated through a model due to the model fitting procedure. Parameter uncertainty arises due to non-unique parameter values that the modeler must choose from, as well the large number of parameters typically needed for ecosystem models.\r\n\r\nThe EwE approach provides some means to execute parameter uncertainty analysis (e.g., (Coll and Steenbeek, 2017; Heymans et al., 2016; Steenbeek et al., 2018). As the Ecospace module inherits model structure, parameters, and associated uncertainty from both Ecopath and Ecosim, we recommend using a formal fitting routine and sensitivity analysis in Ecopath and Ecosim (e.g., (Scott et al., 2016) before tackling uncertainty in Ecospace. To assess Ecospace model sensitivity due to the underlying parameterization of Ecopath \/ Ecosim, the Ecosampler module described in section 3.2 can be used (Steenbeek et al., 2018). The Ecoranger module in EwE 5 was a pseudo-Bayesian approach to search for parameters (Christensen et al., 2005; Christensen and Walters, 2004; Walters et al., 1997), though no Bayesian uncertainty analysis tool is currently built into EwE 6. One promising alternative is Bayesian-Species Distribution Models (B-SDM), which implicitly considers uncertainty in spatial distributions and environmental niches. This type of model can be used in a complementary manner with Ecospace to incorporate uncertainty in plausible habitat capacity, response functions or both (see Coll et al., 2019).\r\n\r\nA \u2018bound, search, and score\u2019 generalized approach for tackling uncertainty due to data and parameter uncertainty may be intuitively familiar to most modelers and help to contextualize the various methods, tools, and statistics available to Ecospace users (Fig 9). This simple framework captures the general approach used in a previous ad hoc effort to parameterize Ecospace (Romagnoni et al., 2015).\r\n\r\nThe \u2018bound\u2019 step addresses issues related to computational cost associated with running Ecospace, a typical challenge with the complexity of most ecosystem models. In Ecospace, this must currently be done manually. We recommend selecting Ecospace parameters for sensitivity analysis based on: (a) importance to the questions, patterns, or processes being investigated, (b) those that are most frequently tuned, and (c) computational limitations and model run time. Common non-spatial parameters used to tune Ecospace models are the \u201cvulnerability exchange rate\u201d, \u201cbase dispersal rate\u201d, \u201crelative vulnerability to predation in bad habitat\u201d, and \u201ceffective power of fishing fleets\u201d. Key spatial parameters (those that vary between raster cells) may also be bounded using the spatial-temporal framework by importing raster data prepared in advance that represent habitat capacity (Coll et al., 2019; Steenbeek et al., 2013).\r\n\r\nIn Ecospace, the \u2019search\u2019 step is currently limited to a manual search, since Ecosampler and the Monte Carlo routine are not yet able to access Ecospace parameters. However, the spatial-temporal framework makes it possible to easily run and re-run Ecospace using different spatial parameterizations, and the ability to run EwE directly from code greatly facilitates the repeated execution of model exercises to perform structural uncertainty analyses.\r\n\r\n\u2018Scoring\u2019 the Ecospace runs may be done in several ways. A simple approach is to use comparisons between Ecospace results from multiple runs. Currently, Spearman\u2019s spatial correlation, which can be calculated using the \u2018raster\u2019 package in R using the \u201ccorLocal\u201d function (R Core Team, 2018) is popular (Coll et al., 2019; Piroddi et al., 2022; Romagnoni et al., 2015; Subramaniam et al., 2022), while (Lynam et al., 2017) used Mantel spatial correlation with 999 permutations to test the significance level. To best address the task of comparing Ecospace outputs, a spatial-temporal statistical measure of similarity, or spatial-temporal cross correlation, is required. Unfortunately, spatial cross-correlation statistics in Ecospace are still under development and present computational and theoretical challenges (Chen, 2015). In the lack of such statistics, P\u00fcts et al. (2020) utilized a set of metrics for the temporal (Pearson correlation and root mean squared error) and spatial fit (Schoener's D index) separately, finally combining them using a Taylor diagram.\r\n\r\n[caption id=\"\" align=\"aligncenter\" width=\"1324\"]\"image\" Figure 9<\/strong> - Diagram of the \u2018bound, search, and score\u2019 framework for tackling uncertainty which captures the general approaches used to parameterize Ecospace[\/caption]\r\n\r\nSpatially-temporally explicit uncertainty assessments are a challenging priority for future development (Steenbeek et al., 2021). The development plans for EwE includes extending the parameters that can be perturbed by the Monte Carlo routine (and subsequently sampled by Ecosampler) to Ecospace (Steenbeek et al., 2021, 2018). Given the high computational cost of running a spatially resolved model such as Ecospace, the parameter search space (i.e. the range of the parameters to be explored, and their combinations) must be limited to a local instead of a global search (Hansen et al., 2019; Pantus, 2007; P\u00fcts et al., 2020), with the caveat of reducing the possibility of exploring extreme values and combination of parameters. Iterative, ad-hoc process for space reduction (e.g., see Romagnoni et al., 2015) are time-consuming when manually implemented, and automated procedures are not available at present. Preliminary understanding of parameter space is thus critical to make the analysis tractable.\r\n

Ecosampler<\/h1>\r\nEcosampler is a module within EwE to measure the impact of Ecopath input parameter sensitivity on its results (Steenbeek et al., 2018). Ecosampler records samples \u2013 alternate mass-balanced parameter sets for a food-web model \u2013 from the built-in Monte Carlo routine, and replays these samples through all of the EwE modules and any loaded plug-in. Ecosampler can capture output variation due to base input parameter sensitivity of any computational component including the temporal module Ecosim, the spatial\u2013temporal module Ecospace, and plug-ins such as Ecological Network Analysis, Value Chain, and Ecological Indicators (Fig. 10).\r\n\r\n[caption id=\"\" align=\"aligncenter\" width=\"1046\"]\"image\" Figure 10<\/strong> - Predicted (solid lines) versus observed (dots) biomass for various functional groups of the model for the Israeli Mediterranean continental shelf ecosystem model for the period 1994\u20132010. Blue shadows represent the 5th and 95th percentiles obtained from Ecosampler-perturbed model outputs. Rho and p-values were obtained from Spearman\u2019s rank correlation test (modified with permission from Corrales et al., 2017)[\/caption]\r\n\r\nImage attributions?<\/span>\r\n
1<\/a> https:\/\/www.un.org\/bbnj\/content\/background<\/u><\/div>\r\n
2<\/a> https:\/\/iaac-aeic.gc.ca\/050\/evaluations\/proj\/80054<\/div>\r\n
3<\/a> https:\/\/ditto-oceandecade.org\/<\/div>\r\n
4<\/a> https:\/\/marine.copernicus.eu\/news\/ocean-and-its-digital-twin-whats-copernicus-marine<\/div>\r\n
\r\n

Adaption<\/strong><\/h1>\r\nThe chapter is in part adapted, with permission,\u00a0 from:\r\n\r\nDe Mutsert K, Marta Coll, Jeroen Steenbeek, Cameron Ainsworth, Joe Buszowski, David Chagaris, Villy Christensen, Sheila J.J. Heymans, Kristy A. Lewis, Simone Libralato, Greig Oldford, Chiara Piroddi, Giovanni Romagnoni, Natalia Serpetti, Michael Spence, Carl Walters. 2023. Advances in spatial-temporal coastal and marine ecosystem modeling using Ecopath with Ecosim and Ecospace<\/a>. Treatise on Estuarine and Coastal Science, 2nd Edition. Elsevier.\r\n\r\n<\/div>\r\n<\/div>","rendered":"
\n

General Practices<\/h1>\n

It is generally a good practice \u2013 if not mandatory \u2013 to explicitly acknowledge sources of uncertainty in data and modeling exercises, make reasonable attempts to quantify it, and caveat the model results as appropriate.<\/p>\n

An ever-increasing body of literature categorizes the various types of uncertainty related to the development and operation of ecosystem models (Cariboni et al., 2007; Harwood and Stokes, 2003; Payne et al., 2016; Spence et al., 2016; Steenbeek et al., 2021). A useful typology is: (1) model \/ structural, (2) stochastic-, (3) data-, and (4) parameter uncertainty (Gregr and Chan, 2015). Model uncertainty occurs due to potential incomplete or inaccurate abstractions embedded in the structure of the model. Ensemble modeling approaches can be an effective approach to better understanding uncertainty due to implicit architecture of a model: multiple models are used, each representing a different method of abstracting and representing the ecosystem, e.g., Scales et al. (2016), Spence et al. (2018), Lotze et al. (2019) and Tittensor et al. (2021). Structural uncertainty can also be explored using a single model by modifying model equations, for example by aggregating\/disaggregating compartments, changing functional relationships and\/or modifying basic function process, e.g., linear fishing mortality (Coll et al., 2020; Hansen et al., 2019; Pantus, 2007). Stochastic uncertainty occurs due to chaotic or \u2018noisy\u2019 process signals, non-stationary patterns and relationships between variables (Deyle et al., 2016; Litzow et al., 2018; Walters, 1987), or uncertainty in processes represented by the model (e.g., recruitment, (Spence et al., 2016). Data uncertainty results from measurement and observation error and can be propagated through a model due to the model fitting procedure. Parameter uncertainty arises due to non-unique parameter values that the modeler must choose from, as well the large number of parameters typically needed for ecosystem models.<\/p>\n

The EwE approach provides some means to execute parameter uncertainty analysis (e.g., (Coll and Steenbeek, 2017; Heymans et al., 2016; Steenbeek et al., 2018). As the Ecospace module inherits model structure, parameters, and associated uncertainty from both Ecopath and Ecosim, we recommend using a formal fitting routine and sensitivity analysis in Ecopath and Ecosim (e.g., (Scott et al., 2016) before tackling uncertainty in Ecospace. To assess Ecospace model sensitivity due to the underlying parameterization of Ecopath \/ Ecosim, the Ecosampler module described in section 3.2 can be used (Steenbeek et al., 2018). The Ecoranger module in EwE 5 was a pseudo-Bayesian approach to search for parameters (Christensen et al., 2005; Christensen and Walters, 2004; Walters et al., 1997), though no Bayesian uncertainty analysis tool is currently built into EwE 6. One promising alternative is Bayesian-Species Distribution Models (B-SDM), which implicitly considers uncertainty in spatial distributions and environmental niches. This type of model can be used in a complementary manner with Ecospace to incorporate uncertainty in plausible habitat capacity, response functions or both (see Coll et al., 2019).<\/p>\n

A \u2018bound, search, and score\u2019 generalized approach for tackling uncertainty due to data and parameter uncertainty may be intuitively familiar to most modelers and help to contextualize the various methods, tools, and statistics available to Ecospace users (Fig 9). This simple framework captures the general approach used in a previous ad hoc effort to parameterize Ecospace (Romagnoni et al., 2015).<\/p>\n

The \u2018bound\u2019 step addresses issues related to computational cost associated with running Ecospace, a typical challenge with the complexity of most ecosystem models. In Ecospace, this must currently be done manually. We recommend selecting Ecospace parameters for sensitivity analysis based on: (a) importance to the questions, patterns, or processes being investigated, (b) those that are most frequently tuned, and (c) computational limitations and model run time. Common non-spatial parameters used to tune Ecospace models are the \u201cvulnerability exchange rate\u201d, \u201cbase dispersal rate\u201d, \u201crelative vulnerability to predation in bad habitat\u201d, and \u201ceffective power of fishing fleets\u201d. Key spatial parameters (those that vary between raster cells) may also be bounded using the spatial-temporal framework by importing raster data prepared in advance that represent habitat capacity (Coll et al., 2019; Steenbeek et al., 2013).<\/p>\n

In Ecospace, the \u2019search\u2019 step is currently limited to a manual search, since Ecosampler and the Monte Carlo routine are not yet able to access Ecospace parameters. However, the spatial-temporal framework makes it possible to easily run and re-run Ecospace using different spatial parameterizations, and the ability to run EwE directly from code greatly facilitates the repeated execution of model exercises to perform structural uncertainty analyses.<\/p>\n

\u2018Scoring\u2019 the Ecospace runs may be done in several ways. A simple approach is to use comparisons between Ecospace results from multiple runs. Currently, Spearman\u2019s spatial correlation, which can be calculated using the \u2018raster\u2019 package in R using the \u201ccorLocal\u201d function (R Core Team, 2018) is popular (Coll et al., 2019; Piroddi et al., 2022; Romagnoni et al., 2015; Subramaniam et al., 2022), while (Lynam et al., 2017) used Mantel spatial correlation with 999 permutations to test the significance level. To best address the task of comparing Ecospace outputs, a spatial-temporal statistical measure of similarity, or spatial-temporal cross correlation, is required. Unfortunately, spatial cross-correlation statistics in Ecospace are still under development and present computational and theoretical challenges (Chen, 2015). In the lack of such statistics, P\u00fcts et al. (2020) utilized a set of metrics for the temporal (Pearson correlation and root mean squared error) and spatial fit (Schoener’s D index) separately, finally combining them using a Taylor diagram.<\/p>\n

\"image\"
Figure 9<\/strong> – Diagram of the \u2018bound, search, and score\u2019 framework for tackling uncertainty which captures the general approaches used to parameterize Ecospace<\/figcaption><\/figure>\n

Spatially-temporally explicit uncertainty assessments are a challenging priority for future development (Steenbeek et al., 2021). The development plans for EwE includes extending the parameters that can be perturbed by the Monte Carlo routine (and subsequently sampled by Ecosampler) to Ecospace (Steenbeek et al., 2021, 2018). Given the high computational cost of running a spatially resolved model such as Ecospace, the parameter search space (i.e. the range of the parameters to be explored, and their combinations) must be limited to a local instead of a global search (Hansen et al., 2019; Pantus, 2007; P\u00fcts et al., 2020), with the caveat of reducing the possibility of exploring extreme values and combination of parameters. Iterative, ad-hoc process for space reduction (e.g., see Romagnoni et al., 2015) are time-consuming when manually implemented, and automated procedures are not available at present. Preliminary understanding of parameter space is thus critical to make the analysis tractable.<\/p>\n

Ecosampler<\/h1>\n

Ecosampler is a module within EwE to measure the impact of Ecopath input parameter sensitivity on its results (Steenbeek et al., 2018). Ecosampler records samples \u2013 alternate mass-balanced parameter sets for a food-web model \u2013 from the built-in Monte Carlo routine, and replays these samples through all of the EwE modules and any loaded plug-in. Ecosampler can capture output variation due to base input parameter sensitivity of any computational component including the temporal module Ecosim, the spatial\u2013temporal module Ecospace, and plug-ins such as Ecological Network Analysis, Value Chain, and Ecological Indicators (Fig. 10).<\/p>\n

\"image\"
Figure 10<\/strong> – Predicted (solid lines) versus observed (dots) biomass for various functional groups of the model for the Israeli Mediterranean continental shelf ecosystem model for the period 1994\u20132010. Blue shadows represent the 5th and 95th percentiles obtained from Ecosampler-perturbed model outputs. Rho and p-values were obtained from Spearman\u2019s rank correlation test (modified with permission from Corrales et al., 2017)<\/figcaption><\/figure>\n

Image attributions?<\/span><\/p>\n

1<\/a> https:\/\/www.un.org\/bbnj\/content\/background<\/u><\/div>\n
2<\/a> https:\/\/iaac-aeic.gc.ca\/050\/evaluations\/proj\/80054<\/div>\n
3<\/a> https:\/\/ditto-oceandecade.org\/<\/div>\n
4<\/a> https:\/\/marine.copernicus.eu\/news\/ocean-and-its-digital-twin-whats-copernicus-marine<\/div>\n
\n

Adaption<\/strong><\/h1>\n

The chapter is in part adapted, with permission,\u00a0 from:<\/p>\n

De Mutsert K, Marta Coll, Jeroen Steenbeek, Cameron Ainsworth, Joe Buszowski, David Chagaris, Villy Christensen, Sheila J.J. Heymans, Kristy A. Lewis, Simone Libralato, Greig Oldford, Chiara Piroddi, Giovanni Romagnoni, Natalia Serpetti, Michael Spence, Carl Walters. 2023. Advances in spatial-temporal coastal and marine ecosystem modeling using Ecopath with Ecosim and Ecospace<\/a>. Treatise on Estuarine and Coastal Science, 2nd Edition. Elsevier.<\/p>\n<\/div>\n<\/div>\n","protected":false},"author":1909,"menu_order":16,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"chapter-type":[],"contributor":[],"license":[],"class_list":["post-511","chapter","type-chapter","status-publish","hentry"],"part":495,"_links":{"self":[{"href":"https:\/\/pressbooks.bccampus.ca\/eweguide\/wp-json\/pressbooks\/v2\/chapters\/511","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pressbooks.bccampus.ca\/eweguide\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/pressbooks.bccampus.ca\/eweguide\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/eweguide\/wp-json\/wp\/v2\/users\/1909"}],"version-history":[{"count":12,"href":"https:\/\/pressbooks.bccampus.ca\/eweguide\/wp-json\/pressbooks\/v2\/chapters\/511\/revisions"}],"predecessor-version":[{"id":1406,"href":"https:\/\/pressbooks.bccampus.ca\/eweguide\/wp-json\/pressbooks\/v2\/chapters\/511\/revisions\/1406"}],"part":[{"href":"https:\/\/pressbooks.bccampus.ca\/eweguide\/wp-json\/pressbooks\/v2\/parts\/495"}],"metadata":[{"href":"https:\/\/pressbooks.bccampus.ca\/eweguide\/wp-json\/pressbooks\/v2\/chapters\/511\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/pressbooks.bccampus.ca\/eweguide\/wp-json\/wp\/v2\/media?parent=511"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/eweguide\/wp-json\/pressbooks\/v2\/chapter-type?post=511"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/eweguide\/wp-json\/wp\/v2\/contributor?post=511"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/pressbooks.bccampus.ca\/eweguide\/wp-json\/wp\/v2\/license?post=511"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}