Many-task Computing: Bridging the Gap Between High Throughput Computing and High Performance Computing

Many-task computing (MTC) aims to bridge the gap between two paradigms, high-throughput computing (HTC) and high-performance computing (HPC). MTC is reminiscent to HTC, but it differs in the emphasis of using many computing resources over short periods of time to accomplish many computational tasks, where the primary metrics are measured in seconds, not operations per month. MTC denotes high-performance computations comprising of multiple distinct activities, coupled via file system operations. Tasks may be small or large, uniprocessor or multiprocessor, compute-intensive or data-intensive, static or dynamic, homogeneous or heterogeneous. The aggregate number of tasks, quantity of computing, and volumes of data may be extremely large. MTC includes loosely coupled applications that are generally communication-intensive but not naturally expressed using message passing interface commonly found in HPC, drawing attention to the many computations that are heterogeneous but not ?happily? parallel. This book explores fundamental issues in defining the MTC paradigm, as well as theoretical and practical issues in supporting compute and data intensive applications on large scale systems.