Sidharth Kumar

Assistant Professor at UAB CS
sid14@uab.edu

My research lies at the intersection of HPC and Data Analytics. More broadly, I work on problems in big data processing, parallel I/O, scalable algorithms, large data file formats, interactive techniques, scientific visualization, machine learning and performance modeling.

I am looking for students!
More Information

Sidharth Kumar

Publications Education Projects Teaching

PIDX: Scalable I/O for Massive Simulations

The increase in computational power of supercomputers is enabling complex scientific phenomena to be simulated at ever-increasing resolution and fidelity. With these simulations routinely producing large volumes of data, performing efficient I/O at this scale has become a very difficult task. Large-scale parallel writes are challenging due to the complex interdependencies between I/O middleware and hardware. Analytic-appropriate reads are traditionally hindered by bottlenecks in I/O access. Moreover, the two components of I/O, data generation from simulations (writes) and data exploration for analysis and visualization (reads), have substantially different data access requirements. Parallel writes, performed on supercomputers, often deploy aggregation strategies to permit large-sized contiguous access. Analysis and visualization tasks, usually performed on computationally modest resources, require fast access to localized subsets or multiresolution representations of the data. This work tackles the problem of parallel I/O while bridging the gap between large-scale writes and analytics-appropriate reads.



PIDX Pipeline


The focus of this project is to develop an end-to-end adaptive-resolution data movement framework that provides efficient I/O, while supporting the full spectrum of modern HPC hardware. This is achieved by developing technology for highly scalable and tunable parallel I/O, applicable to both traditional parallel data formats and multiresolution data formats, which are directly appropriate for analysis and visualization. To demonstrate the efficacy of the approach, a novel library, PIDX is developed that is highly tunable and capable of adaptive-resolution parallel I/O to a multiresolution data format. Adaptive resolution storage and I/O, which allows subsets of a simulation to be accessed at varying spatial resolutions, can yield significant improvements to both the storage performance and I/O time. The library provides a set of parameters that controls the storage format and the nature of data aggregation across the network; further, a machine learning-based model is constructed that tunes these parameters for the maximum throughput. This work is empirically demonstrated by showing parallel I/O scaling up to 768K cores within a framework flexible enough to handle adaptive resolution I/O.

PIDX deployements


PIDX Pipeline

PIDX Salient features:
  • Has shown scalibility to 768K cores.
  • Generates analysis and visualization appropriate data format.
  • Allows real-time monitoring of simulation.
  • Allows ROI based dumps enabling simulations to dump regions to store data at varying resolution.
  • Uses a compression scheme that allows reading and decompressing data at different bit rates.
  • Has been tested on some of the fastest supercomputers of the world.
  • Exposes lots of parameter and is very tunable.
PIDX enabled visualization


PIDX Pipeline

Combustion simulation data [~300 Terabytes] generated by PIDX visualized using Ospray



Performance Modeling and Autotuning


PIDX Pipeline

HPC systems vary widely in terms of architecture, network topology, and processing capabilities. They also use different file systems and storage hardware that behave differently in response to I/O tuning parameters. Although many of these properties can be empirically determined through characterization studies, it is still difficult to translate them into an optimal set of tuning parameters. My work address this challenge by constructing a machine-learning based performance model to aid in exploration of the parameter space. Moreover, the model can be used to suggest promising parameter configurations and auto-tune the system (see figure). We begin by performing a detailed characterization study of our data system, which is then used as a training dataset to build a performance model. Due to cost restrictions, training of models is confined to experiments conducted over a small number of processes. We obtain samples from the high-core-count regime by augmenting the regression model with a sampling technique. Consequently, we obtain a new model that adaptively improves itself over multiple simulation time steps, autotuning the system. With this work we successfully demonstrate the efficacy of a modeling framework in approximating the behavior of a system at higher core count regimes where brute-force sensitivity studies would be costly and resource intensive






Adaptive-resolution I/O


PIDX Pipeline


With simulations increasing in size and complexity, I/O and storage systems have difficulty keeping pace with the increasing deluge of data that scientists need to generate and store. Increasing I/O time and storage costs leads to a curtailment in the frequency of analysis and checkpoint dumps and also slows down postprocessing analysis tasks. My work addresses this challenge by formally introducing a data movement framework designed specially for adaptive and sparse I/O. The framework transforms the typical dense multidimensional array output into a light-weight sparse representation that can then be written instead of storing the data in their entirety. An example of this approach is region-of-interest (ROI) based data writes that allow regions/subsets of a simulation’s domain to be written at varying resolutions (see figure). The approach led to a significant improvement in both the I/O time as well the storage footprint of the simulation. This work has also been extended to support adaptive-resolution meshes generated from adaptive mesh refined simulations (AMR).