Multi-criteria assessment of ecological process models using pareto optimization

Abstract

Assessment is the essential step in using an ecological process model as a heuristic for investigating hypotheses. Assessment investigates the model's capacity to adequately simulate the phenomenon, as represented by selected criteria. A model's inability to satisfy all assessment criteria simultaneously reveals inadequacies in either the assessment decisions--the criteria formulations or parameter space search, or the model structure--the mathematical representations or the model's collection of underlying hypotheses. An assessment procedure must be able not only to detect each type of deficiency but to distinguish between them, guiding model revision by locating each deficiency's source. Multiple criteria increase an assessment's capacity to do this. Currently there are no multiple criteria model assessment techniques designed both to detect and locate these different types of deficiencies.The Pareto Optimal Model Assessment technique introduced here retains the multiple criteria as a vector rather than aggregating them into a single measure of performance. Optimizing the vector of criteria, generating the Pareto Optimal Set, may reveal that the model requires different parameterizations to satisfy different criteria, can not satisfy a particular criterion with any parameterization, or requires unrealistic parameterizations to satisfy all criteria simultaneously. Investigation of the Pareto Optimal Set reveals these different deficiencies types, and their sources, whether in the assessment decisions or the two levels of model structure specification.The Pareto Optimal Model Assessment technique is applied to the spatially explicit canopy competition model WHORL using ten criteria, binary interval error measures, and simulated evolution optimization to generate the Pareto Optimal Set. Assessment reveals a deficient mathematical representation of physiological plasticity. Assessment of the revised model reveals an inability to limit the growth rate of the tallest dominant trees to the observed range, as well as poor formulation of two criteria. Application of the Pareto Optimal Model Assessment technique to model development and model comparison is discussed. The revised model, WHORL2, is used to critique the representation of the canopy competition process in models of self-thinning of forest stands.