Mitigating the Impact of Data Movement in Memory-Intensive Applications

Abstract

Modern memory-intensive workloads move terabytes of data through hierarchies of interconnects whose bandwidth and latencies have failed to scale with compute. The result is that there is no longer a singular memory wall between compute and memory, but layers of connections, each a potential bottleneck. Caches, memory bus width, and inter-device interconnects all have an impact on the performance that applications observe. This dissertation posits that overcoming these limitations requires identifying key application characteristics that enable bottleneck mitigation. We identify and organize mitigations into three types: (1) avoiding data movement across certain layers by eliding extraneous or redundant data movement, (2) hiding data movement by overlapping transfers with complementary work, and (3) reducing the amount of data moved by exploiting structure present within the data itself. This characterization identifies the application properties that enable intervention, while directly informing us on how the mitigation should be designed. We discuss three systems—(MC)^2, POD-Attention, and Invariant Bit Packing (IBP)—that demonstrate these concepts. Each system targets a different bottleneck component, taking advantage of application properties that enable the mitigation. They provide simple integration while retaining application correctness. (MC)^2 targets bottlenecks with CPU memory access latencies, exploiting the observation that many workloads copy data that they never fully access. It extends the CPU memory controller to lazily track requested copies. Laziness allows us to avoid copying data that is not accessed, while implementation at the memory controller targets the largest source of copy overhead—stalls on cache misses. POD-Attention targets the GPU memory bandwidth bottleneck in LLM inference, taking advantage of the complementary resource profiles of the two phases of LLM inference, prefill and decode. It fuses compute-bound prefill and memory-bound decode so that they execute concurrently—the prefill operations occupy the compute units while decode operations saturate the memory pipeline, hiding memory stalls behind concurrent compute. IBP targets the PCIe interconnect bandwidth bottleneck between CPU and GPU memory that occurs in large-scale ML frameworks, taking advantage of low-entropy bit patterns present across ML tensors. It strips these bits before PCIe transfer and reconstructs the original tensors within the GPU, reducing data transfer volume without loss of information. These systems achieve substantial benefits across a wide variety of workloads. (MC)^2 targets databases and operating system primitives showing up to 78% higher throughput for database transactions and 250X lower latency on huge page copy-on-write faults. POD-Attention enhances LLM inference providing up to 59% faster LLM attention operations. IBP improves ML offloading, providing on average 74% faster GNN training, 180% faster DLRM embedding lookups, and 24% faster offloaded LLM inference.