Automated Reproducibility
Being able to recreate any run of the software—for testing purposes and to check claims about its outputs—in an automated way (not just via manual recreation from documentation). This includes provenance (and perhaps automated recreation) of the entire computational environment (since results can vary based on things like the versions of external libraries used).
Cohesive, Loosely-Coupled Design
A design separated into components with well-defined (cohesive) functions, and minimised dependencies on other components (loose-coupling). This massively aids the debugging, maintenance and reusability of the code. This often involves reusing recurring structural and behavioural forms that have been shown to help solve common design issues: SE calls such forms design patterns (Gamma et al. 1995; Buschmann 1996). Such forms help establish a shared software design vocabulary at a higher level of abstraction.
Testability
Being designed in a way that facilitates testing at different levels (e.g., single class, component or whole system) and, where possible, includes automated tests as part of the software deliverable. In particular, automated tests provide a bank of regression tests which can be continually re-run to check that changes have not caused bugs elsewhere (i.e., caused previously successful tests to fail). Such tests become the central driver of the development process in the increasingly-used Test-Driven Development (TDD) approach (Jeffries & Melnik 2007).
However, there appears to be virtually no discussion of these issues more generally for 'mainstream' simulation using widely-used toolkits such as, in the ABM case, NetLogo (Tisue & Wilensky 2004), Repast Simphony (North et al. 2013), MASON (Luke et al. 2005), or AnyLogic (Borshchev & Filippov 2004). In particular, there is nothing which allows simulation practitioners to understand how these ideas might be embodied in some best-practice simulation design, and to therefore have some frame to assess existing toolkits and make more informed decisions on their choice of simulation platform (and thus understand the strengths and weaknesses of their simulation software design with respect to this best-practice).
Utilities.
General utilities (not specific to the domain model or upper layers) for (a) data types (e.g., linked lists); (b) input/output capabilities, such as to/from different file formats; and (c) general algorithmic facilities such as random number generators, probability distributions or differential equation numerical solvers. These can interact; e.g., probability distributions could be initialised from external files.
Domain Model.
The code representing the abstraction of the real-world system, including the representation of space and time.
Execution Control.
How the domain model is actually executed, which typically amounts to instantiating a 'root' object and stepping through a schedule of actions (provided by a domain model component) to 'unfold' time dynamically. Because non-domain-model objects also need to interleave their actions in simulated time, this layer includes that capability. This is a 'thin' layer, but nevertheless a well-defined one.
Meta-Data Capture.
Code (scheduled in simulation time) to capture and calculate meta-data; i.e., derived model state (possibly held as a time series to capture changes over time) or atomic model state captured over time.
This layer includes any writing of outputs to file (or database) because this can be tightly coupled with meta-data capture; in larger-scale simulations, time series data may be captured in a rolling window for storage reasons (perhaps with this window used for visualisation), with outputs written to file as they 'drop out of' the window (or via some other buffering strategy).
State & Control Presentation.
The parts of the user interface which present model state and controls as part of a user interface. The presentation may be visual or textual. Where current model state is being presented, this directly uses the Domain Model layer. If meta-data is being presented, this uses the Meta-Data Capture layer.
In particular, note that a given domain model might have multiple presentations, with multiple alternative visualisations per component; such solutions require a layered domain model separation.
Experiments Definition.
The parts of the user interface which support the definition of simulation runs (experiments), possibly including multi-run experiments. This mainly consists of how model inputs are defined and passed on to the model, and any automated manipulation of them across multiple runs for things like sensitivity analysis. Because this tends to be particularly generic to any simulation (and modellers using multiple toolkits may want a vendor-neutral solution), separate experimental platforms exist (Gulyás et al. 2011), and I am aware of simulation consultancies who develop their own in-house.
References
- ALLAN, R. J. (2010). Survey of agent-based modelling and simulation tools. Tech. Rep. DL-TR-2010-007, Science & Technologies Facilities Council (STFC), UK.
BORSHCHEV, A. & Filippov, A. (2004). From system dynamics and discrete event to practical agent based modeling: Reasons, techniques, tools. In: Proceedings of the 22nd International Conference of the System Dynamics Society.
BRAILSFORD, S. C. (2014). Discrete-event simulation is alive and kicking! Journal of Simulation 8(1), 1–8.
BUSCHMANN, F. (1996). Pattern-Oriented Software Architecture: a system of patterns. Wiley, volume 1 ed. http://www.worldcat.org/isbn/0471958697.
COLLIER, N. & Ozik, J. (2013). Test-driven agent-based simulation development. In: Proceedings of the 2013 Winter Simulation Conference (Pasupathy, R., Kim, S. H., Tolk, A., Hill, R. & Kuhl, M. E., eds.). IEEE.
DAVISON, A. P., Mattioni, M., Samarkanov, D. & Teleńczuk, B. (2014). Sumatra: A toolkit for reproducible research. In: Implementing Reproducible Research (Stodden, V., Leisch, F. & Peng, R. D., eds.), chap. 3. Chapman & Hall, pp. 57–78.
DJANATLIEV, A., Dulz, W., German, R. & Schneider, V. (2011). VERITAS—a versatile modeling environment for test-driven agile simulation. In: Proceedings of Wintersim 2011 (Jain, S., Creasey, R. R., Himmelspach, J., White, K. P. & Fu, M., eds.). http://www.informs-sim.org/wsc11papers/325.pdf.
EVANS, E. (2004). Domain-Driven Design: tackling complexity in the heart of software. Addison-Wesley.
EWALD, R. & Uhrmacher, A. M. (2014). SESSL: A domain-specific language for simulation experiments. ACM Trans. Model. Comput. Simul. 24(2). . [doi:10.1145/2567895]
FOWLER, M. (2004). UML Distilled: A Brief Guide to the Standard Object Modeling Language. Addison-Wesley, third ed. http://www.worldcat.org/isbn/0321193687.
FOWLER, M., Rice, D., Foemmel, M., Hieatt, E., Mee, R. & Stafford, R. (2003). Patterns of Enterprise Application Architecture. Addison-Wesley.
GAMMA, E., Helm, R., Johnson, R. & Vlissides, J. (1995). Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley.
GILBERT, N. & Troitzsch, K. G. (2005). Simulation for the Social Scientist. Open University Press, 2nd ed.
GREEN, T. R. G. & Petre, M. (1996). Usability analysis of visual programming environments: A 'cognitive dimensions' framework. Journal of Visual Languages & Computing 7(2), 131–174. . [doi:10.1006/jvlc.1996.0009]
GRIMM, V. & Railsback, S. F. (2005). Individual-based modeling and ecology. Princeton Series in Theoretical and Computational Biology. Princeton University Press. http://www.worldcat.org/isbn/0691096651.
GULYÁS, L., Szabó, A., Legéndi, R., Máhr, T., Bocsi, R. & Kampis, G. (2011). Tools for large scale (distributed) agent-based computational experiments. In: Proceedings of CSSSA 2011. Computational Social Science Society of the Americas (CSSSA).
GÜRCAN, O., Dikenelli, O. & Bernon, C. (2013). A generic testing framework for agent-based simulation models. Journal of Simulation 7(3), 183–201. . [doi:10.1057/jos.2012.26]
HIMMELSPACH, J. & Uhrmacher, A. M. (2007). Plug'n simulate. In: 40th Annual Simulation Symposium (ANSS'07). Washington, DC, USA: IEEE. .
JEFFRIES, R. & Melnik, G. (2007). TDD: The art of fearless programming. IEEE Software 24(3), 24–30. . [doi:10.1109/MS.2007.75]
JOINES, J. A. & Roberts, S. D. (1999). Simulation in an object-oriented world. In: Proceedings of the 1999 Winter Simulation Conference (Farrington, P. A., Nembhard, H. B., Sturrock, D. T. & Evans, G. W., eds.). .
KERNIGHAN, B. (1979). UNIX for Beginners. Bell Telephone Labs, 2nd ed.
LUKE, S., Cioffi-Revilla, C., Panait, L., Sullivan, K. & Balan, G. (2005). MASON: A multiagent simulation environment. Simulation 81(7), 517-527. [doi:10.1177/0037549705058073]
MCCONNELL, S. (2004). Code Complete: A practical handbook of software construction. Microsoft Press, second ed. http://www.worldcat.org/isbn/9780735619678.
MILLER, J. H. & Page, S. E. (2007). Complex Adaptive Systems: An Introduction to Computational Models of Social Life. Princeton Studies in Complexity. Princeton Press.
MILLINGTON, J. D. A., O'Sullivan, D. & Perry, G. L. W. (2012). Model histories: Narrative explanation in generative simulation modelling. Geoforum 43(6), 1025–1034. . [doi:10.1016/j.geoforum.2012.06.017]
MULDER, J. D., van Wijk, J. J. & van Liere, R. (1999). A survey of computational steering environments. Future Generation Computer Systems 15(1), 119–129. . [doi:10.1016/S0167-739X(98)00047-8]
MÜLLER, J. P. (2009). Towards a formal semantics of event-based multi-agent simulations. In: Multi-agent Based Simulation IX, no. 5269 in LNCS. Springer.
NIKOLAI, C. & Madey, G. (2009). Tools of the trade: A survey of various agent based modeling platforms. Journal of Artificial Societies and Social Simulation, 12(2), 2. https://www.jasss.org/12/2/2.html.
NORTH, M. J., Collier, N. T., Ozik, J., Tatara, E. R., Macal, C. M., Bragen, M. & Sydelko, P. (2013). Complex adaptive systems modeling with Repast Simphony. Complex Adaptive Systems Modeling 1(1), 3+. . [doi:10.1186/2194-3206-1-3]
NORTH, M. J. & Macal, C. M. (2014). Product and process patterns for agent-based modelling and simulation. Journal of Simulation, 8, 25-36. . [doi:10.1057/jos.2013.4]
ORAM, A. & Wilson, G. (eds.) (2010). Making software : what really works, and why we believe it. O'Reilly. http://www.worldcat.org/isbn/9780596808327.
RAILSBACK, S. F. & Grimm, V. (2012). Agent-based and individual-based modeling : a practical introduction. Princeton University Press. http://www.worldcat.org/isbn/9780691136745.
RAILSBACK, S. F., Lytinen, S. L. & Jackson, S. K. (2006). Agent-based simulation platforms: review and development recommendations. Simulation 82, 609–623. http://www.humboldt.edu/ecomodel/documents/ABMPlatformReview.pdf. [doi:10.1177/0037549706073695]
ROPELLA, G. E., Railsback, S. F. & Jackson, S. K. (2002). Software engineering considerations for individual-based models. Natural Resource Modeling 15(1), 5–22. [doi:10.1111/j.1939-7445.2002.tb00077.x]
ROUCHIER, J., Cioffi-Revilla, C., Polhill, J. G. & Takadama, K. (2008). Progress in model-to-model analysis. Journal of Artificial Societies and Social Simulation 11(2), 8. https://www.jasss.org/11/2/8.html.
SANDVE, G. K., Nekrutenko, A., Taylor, J. & Hovig, E. (2013). Ten simple rules for reproducible computational research. PLoS Comput Biol 9(10), e1003285+. . [doi:10.1371/journal.pcbi.1003285]
SEGAL, J. (2008). Scientists and software engineers: A tale of two cultures. In: Proceedings of the Psychology of Programming Interest Group PPIG 08.
SOMMERVILLE, I. (2011). Software engineering. Pearson, 9th ed. http://www.worldcat.org/isbn/9780137053469.
STODDEN, V., Donoho, D., Fomel, S., Freidlander, M. P., Gerstein, M., Leveque, R., Mitchell, I., Larrimore Ouellette, L. & Wiggins, C. (2010). Reproducible research: Addressing the need for data and code sharing in computational science. Computing in Science & Engineering 12(5), 8–13. .
STODDEN, V., Guo, P. & Ma, Z. (2013). Toward reproducible computational research: An empirical analysis of data and code policy adoption by journals. PLoS ONE 8(6), e67111+. . [doi:10.1371/journal.pone.0067111]
TISUE, S. & Wilensky, U. (2004). NetLogo: Design and implementation of a multi-agent modeling environment. In: Proceedings of Agent 2004.
UHRMACHER, A. M. (2012). Seven pitfalls in modeling and simulation research. In: Proceedings of the 2012 Winter Simulation Conference (Laroque, C., Himmelspach, J., Pasupathy, R., Rose, O. & Uhrmacher, A. M., eds.).
VIANA, J., Rossiter, S., Channon, A. A., Brailsford, S. C. & Lotery, A. (2012). A multi-paradigm, whole system view of health and social care for age-related macular degeneration. In: Proceedings of the Winter Simulation Conference, WSC '12. Winter Simulation Conference. http://dl.acm.org/citation.cfm?id=2429759.2429884.
WHITLEY & Blackwell, A. F. (2001). Visual programming in the wild: A survey of LabVIEW programmers. Journal of Visual Languages & Computing 12(4), 435–472. . [doi:10.1006/jvlc.2000.0198]
WILSON, G. (2014). Software carpentry: lessons learned. F1000Research . [doi:10.12688/f1000research.3-62.v1]
WILSON, G., Aruliah, D. A., Brown, C. T., Chue Hong, N. P., Davis, M., Guy, R. T., Haddock, S. H. D., Huff, K. D., Mitchell, I. M., Plumbley, M. D., Waugh, B., White, E. P. & Wilson, P. (2014). Best practices for scientific computing. PLoS Biol 12(1), e1001745+. . [doi:10.1371/journal.pbio.1001745]
ZEIGLER, B. P., Gon Kim, T. & Praehofer, H. (2000). Theory of modeling and simulation : integrating discrete event and continuous complex dynamic systems. Academic Press, 2nd ed.
ZINN, S., Himmelspach, J., Uhrmacher, A. M. & Gampe, J. (2013). Building Mic-Core, a specialized M&S software to simulate multi-state demographic micro models, based on JAMES II, a general M&S framework. Journal of Artificial Societies and Social Simulation 16(3), 5. https://www.jasss.org/16/3/5.html.
https://www.mathworks.com/company/newsletters/articles/using-modeling-and-simulation-to-test-designs-and-requirements.html
0 Comments