Effective Program Analyses for Automated Software Testing and Error Diagnosis

Abstract

This dissertation presents five program analysis techniques for improving automated software testing and error diagnosis. Two techniques aim to help software developers build reliable software and three techniques assist software users in quickly identifying the root cause of an exhibited software error. The first two techniques help software developers validate software's behaviors before delivering it to users. The first technique, called Palus, assists developers in creating unit test suites for software systems with constrained interfaces. Palus combines dynamic model inference, static analysis, and random testing to automatically generate unit tests that are legal and behaviorally-diverse. Tests generated by Palus achieve higher code coverage, and detect more bugs than tests generated by existing techniques. When a test fails, a developer must understand the cause of the failure before fixing the bug. The second technique, called FailureDoc, helps developers interpret the test results. FailureDoc uses static analysis, runtime monitoring, and statistical reasoning to augment a failed test with contextually-relevant debugging clues: code comments that provide useful facts about the failure, and assists developers in understanding why a test fails. The other three techniques help software users diagnose exhibited software errors. The third technique, called ConfDiagnoser, assists software users in troubleshooting configuration errors. It uses static analysis, dynamic profiling, and statistical analysis to link the undesired behavior to specific configuration options whose values should be changed. The fourth technique, called ConfSuggester, extends ConfDiagnoser to support configuration error diagnosis for evolving software systems. ConfSuggester uses the desired behavior of the old software version as a baseline against which to compare new program behavior, and reasons about the root-cause configuration option. The fifth technique, called FlowFixer, helps users repair UI workflow errors. Given a new software version's changed UI, in which a user's desired action is not possible, FlowFixer uses dynamic profiling, static analysis, and random testing to inform users of replacement UI actions to fix a broken workflow. These techniques have been implemented and evaluated on real-world software systems. The experimental results and user studies have shown that the implemented techniques are accurate, efficient, and practically useful.