Supporting bioinformatics analysis using a hybrid cloud and HPC architecture

Abstract

The exponential growth of next-generation sequencing data requires novel strategies for storage, transfer, and processing of said data. We present a scheduler a based on the Temporal.io workflow framework which enables two key optimizations of bioinformatics workflows. Firstly, we enable users to transparently map workflow steps to diverse execution environments, including high-performance computing (HPC) resources managed by the SLURM resource manager. When tested on a Bulk RNA sequencing workflow, this feature allows a 26% reduction in credit consumption on the NSF Bridges 2 supercomputer by performing adapter trimming locally and all other steps on the supercomputer. Secondly, we enable asynchronous execution of workflows, a feature which guarantees that workflows will achieve reasonable resource utilization even when the scheduler cannot make use of a system's full RAM and CPU resources. When benchmarked on the same Bulk RNA sequencing workflow, this optimization facilitates a reduction in workflow makespan of between 13% and 23%, depending on the exact workflow configuration. Taken together, these features will enable reductions in the cost and time requirements of bioinformatics pipelines for researchers.