Failure Diagnosis for Datacenter Applications

Abstract

Fast and accurate failure diagnosis remains a major challenge for datacenter operators. Current datacenter applications are increasingly architected around loosely-coupled modular components: each component can scale and evolve independently. However, when application failures occur, they become much harder to detect and localize. The challenges are three-fold: complex component dependency, gray failures, and unpredictable component behaviors. My thesis is that fast and accurate failure diagnosis for datacenter applications is possible using three key ideas: (1) a global view of component interactions and dependencies, (2) a penalized-regression-based failure localization algorithm that localizes both fail-stop and gray failures, and (3) a network architecture that produces predictable routes, simplifying failure localization without sacrificing load balancing and other network features. I present two complementary systems to demonstrate this. The first, Deepview, is a system that can localize virtual hard disk (VHD) failures in Infrastructure-as-a-Service clouds. I show that Deepview localizes VHD failures accurately and quickly to compute, storage and network components in production at Microsoft Azure. The second, Volur, is a network architecture that makes in-network routing predictable to the end-hosts. I show that Volur accurately localizes non-fail-stop link or switch failures and approximates state-of-the-art dynamic load balancing schemes.