Information Artifact Evaluation and Iterative Design: A Novel Mixed-Method Approach

Abstract

Organizations face numerous design challenges regarding effectiveness and measurability of success when investing in Information Systems and Information Technology (IS/IT), or information artifacts for short. Some information artifacts produce better informational and transactional results than others, but it is not clearly understood which information artifacts do better, and why. The designing, building, maintaining, and upgrading of information artifacts needs to be better informed, more targeted, and predictably effective. This requires a deeper understanding of context-sensitive artifact evaluation criteria and their link to design choices. Combining qualitative and quantitative evaluation and outcome testing methods promises to be an innovative way to improve artifacts iteratively. The contribution of this study is twofold: It provides an empirical examination and validation of an iterative and phased assessment approach, and it introduces a novel methodological approach. Through empirical testing, qualification, and validation, the results demonstrate that information artifacts can be targeted and improve. This has great relevance to information artifact evaluation and design practice in any organization of any size and any sector. The study employs the TEDS assessment framework and assessment methodology to evaluate and rank 10 iOS mobile sports applications in eight use case scenarios. A design method is developed to leverage the TEDS rater output of screenshots and comments from the evaluations. Through brainstorming, design patterns are identified based on the highest performing assessed applications and used to assist directive treatment development. Feature treatments are created and used in A/B testing by the organization of one of the 10 assessed iOS mobile applications. Upon completion of the A/B testing phase and treatment deployment, a subsequent TEDS evaluation of the application was performed that showed improved ratings for the application.