Critical Mass: Performance and Programmability Evaluation of MASS (Multi-Agent Spatial Simulation) and Hybrid OpenMP/MPI

Abstract

In this paper, we explore the relationship between programmability and performance within the context of two C++ parallel/distributed programming approaches: Hybrid OpenMP/MPI & MASS (Multi-Agent Spatial Simulation). Our study begins by working with the following hypothesis: programmers in big data analysis and Agent-Based Models (ABM) will find MASS easier to use than hybrid OpenMP/MPI, despite its slower performance. We then detail the planned experiments and criteria used for testing this hypothesis, which include a mixture of broadly-accepted characteristics for programmability within parallel/distributed frameworks, survey application, line of code counting, and actual performance testing. During our research, we found that MASS offered more of a global view of computation than hybrid OpenMP/MPI and that programmers typically took 39 minutes less to write corresponding applications using MASS. When writing these applications, MASS required around 8.17% less parallel/distributed-specific lines of code. In addition, we learned that applications written in MASS were approximately 4.4% easier to debug than corresponding ones based on OpenMP/MPI. While there were promising results for MASS, our data showed that OpenMP/MPI slightly outperformed MASS in general characteristics of programmable parallel/distributed frameworks and received more favorable assessments across most surveyed questions related to time, effort, and programmability. We also found that the same application written in OpenMP/MPI typically had an execution time that was 25.82% better (lower) than corresponding applications built using MASS. Overall, even though we found that the programmability results were quite close between the two frameworks, we were unable to accept the alternative hypothesis presented. It is worth noting, however, that the C++ version of MASS is around 3 years old and is actively being developed by a small handful of students and faculty at the University of Washington Bothell. Whereas, OpenMP/MPI has nearly two decades of development and support from major hardware/software corporations across the world.