Scalable Tools Workshop

Pipit: Enabling Programmatic Analysis of Parallel Execution Traces

Abhinav Bhatele

Abstract

Performance analysis is an important part of the oft-repeated, iterative process of performance tuning during the development of parallel programs. Per-process per-thread traces (detailed logs of events with timestamps) enable in-depth analysis of parallel program execution to identify various kinds of performance issues. Often times, trace collection tools provide a graphical tool to analyze the trace output. However, these GUI-based tools only support specific file formats, are difficult to scale when the data is large, limit data exploration to the implemented graphical views, and do not support automated comparisons of two or more datasets. In this talk, we present a programmatic approach to analyzing parallel execution traces by leveraging pandas, a powerful Python-based data analysis library. We have developed a Python library, Pipit, on top of pandas that can read traces in different file formats (OTF2, HPCToolkit, Projections, Nsight, etc.) and provide a uniform data structure in the form of a pandas DataFrame. Pipit provides operations to aggregate, filter, and transform the events in a trace to present the data in different ways. We also provide several functions to quickly identify performance issues in parallel executions.

Rapid and Effective Load-Level Memory Trace Analysis

Nathan Tallent

Abstract

A challenge of memory trace analysis is combining detailed analysis and low overhead measurement. Currently, hardware/software-based analysis of load-level sequences easily incurs time slowdowns of 100×. We will discuss MemGaze, a tool for low-overhead, high-resolution memory trace analysis. MemGaze uses Intel's Processor Tracing (PT) instruction ptwrite to collect sampled and compressed memory address traces for load-level, sequence-aware analysis of data reuse. We describe multi-resolution analysis for locations vs. operations, accesses vs. spatio-temporal reuse, and reuse (distance, rate, volume) vs. access patterns. Both trace size and resolution are controllable.

DeepProf: Interpreting Performance Profiles with Deep Learning

Qidong Zhao

Abstract

Nowadays, profilers have been adopted as part of the software stack in most system environments, ranging from embedded systems and laptops to supercomputers and clouds. However, even for experts, leveraging profilers to identify actionable optimization opportunities in program source code is challenging. The main reason is that profiles are typically disconnected from program semantics; users need substantial manual efforts to map profiles to program source code and identify actionable optimization opportunities for source code refactoring.

In this talk, we introduce DeepProf, the first end-to-end tool that combines performance profiles and program semantics with a deep learning approach. Our key idea is to associate code summary, which is obtained from deep learning models, with profiles to understand performance inefficiencies better. DeepProf supports multiple profilers, various programming languages, and different deep learning models. We evaluate DeepProf with microbenchmarks and production applications, showing that DeepProf can give an extra dimension of information to facilitate performance analysis and optimization. As a practical tool, DeepProf has been implemented as a Visual Studio Code (vscode) extension, which can be directly installed via the vscode marketplace.

E4S

Sameer Shende

Abstract

The Extreme-scale Scientific Software Stack (E4S) [https://e4s.io] is a curated, Spack based software distribution of 100+ HPC and AI/ML packages. The Spack package manager is a core component of E4S and it is a platform for product integration and deployment of performance evaluation tools such as TAU, HPCToolkit, DyninstAPI, PAPI, etc. and supports both bare-metal and containerized deployment for CPU and GPU platforms. E4S provides a Spack binary cache and a set of base and full-featured container images with vendor runtimes to support GPU architectures from NVIDIA, Intel, and AMD. E4S is a community effort to provide open-source software packages for developing, deploying, and running scientific applications and tools on HPC platforms. It has built a comprehensive, coherent software stack that enables application developers to productively develop highly parallel applications that effectively target diverse exascale architectures. E4S supports commercial cloud platforms, including AWS with multiple MPI implementations and containers with a Remote Desktop DCV application that simplifies performance tool integration and deployment. It also includes a container launch tool (e4s-cl) that allows binary distribution of applications by substituting MPI in the containerized application with the system MPI. It features tools to customize base container images provided by E4S by adding packages using Spack and OS package managers. These containers are available for download from the E4S website and DockerHub. This talk will describe performance tool and runtime systems integration issues with GPU runtimes in E4S and instrumentation APIs exposed for performance evaluation tools.

Bio:

Sameer Shende serves as a Research Professor and the Director of the Performance Research Laboratory at the University of Oregon and the President and Director of ParaTools, Inc. (USA) and ParaTools, SAS (France). He serves as the lead developer of the Extreme-scale Scientific Software Stack (E4S), TAU Performance System, Program Database Toolkit (PDT), and HPC Linux. His research interests include scientific software stacks, performance instrumentation, compiler optimizations, measurement, and analysis tools for HPC. He leads the SDK project for the Exascale Computing Project (ECP), in the Programming Models and Runtime (PMR) area.

Thicket: Seeing the Performance Experiment Forest for the Individual Run Trees

Stephanie Brink

Abstract

Thicket is an open-source Python toolkit for Exploratory Data Analysis (EDA) of multi-run performance experiments. It enables an understanding of optimal performance configuration for large- scale application codes. Most performance tools focus on a single execution (e.g., single platform, single measurement tool, single scale). Thicket bridges the gap to convenient analysis in multi- dimensional, multi-scale, multi-architecture, and multi-tool performance datasets by providing an interface for interacting with the performance data. We demonstrate the power and flexibility of Thicket through two case studies, first with the open-source RAJA Performance Suite on CPU and GPU clusters and another with a large physics simulation run on both a traditional HPC cluster and an AWS Parallel Cluster instance.

Acceptance Testing with Pavilion

Paul Ferrell

Abstract

Testing new clusters for operability, performance, and contract compliance, otherwise known as acceptance testing, generally requires a great deal of time and effort from a significant number of staff. Tests must be scripted, made compatible with the new cluster, and calibrated for optimal performance. Documentation for running these tests is typically lacking, with expertise spread across disparate individuals who each have their own methodologies and organization. Everything from test build scripts to results are in custom, incompatible formats, necessitating a substantial amount of coordination when performing acceptance testing.

Pavilion 2.0 was developed with the goal of streamlining the day-to-day operational and regression testing of production clusters at Los Alamos National Laboratory (LANL). Pavilion provides a standardized, YAML config based method which automatically handles many of the complicated and system dependent tasks involved in building and running tests and performance metrics. Results are gathered automatically into a common JSON format which can be displayed or graphed directly via Pavilion, or fed into Splunk or similar tools. Pavilion has allowed our team to drastically improve the quality and quantity of the tests we run to ensure consistent functionality of our systems.

With the advent of Crossroads, our new flagship Advanced Technology System (ATS) and its associated smaller clusters, we sought to simplify our acceptance testing processes as well. While this has involved the addition of new features in Pavilion itself, it has primarily entailed the development of Pavilion test configs for a wide variety of performance and functionality tests. By developing the test configurations under Pavilion, we were able to take advantage of Pavilion’s ability to modularize the process, allowing these tests to be easily adapted to each of the new clusters. Moreover, the standardization under Pavilion makes it so that anyone on our testing team can now build, run, and gather standardized results for any of the acceptance tests. These new test configurations were written with more than just LANL in mind - they are intended for incorporation into Pavilion itself as a library adaptable to any new cluster.

Score-P: Beyond the Infinite HPC domain

Bert Wesarg, Bill Williams

Abstract

Score-P is a well-established framework for instrumentation and measurement of parallel codes. It has its origin in the classic HPC field. The HPC field is characterized by providing tightly coupled communication methods and shared file systems to applications, which Score-P also profits from. These are, and always will be, key features for HPC applications and measurement frameworks to achieve exceptional performance and scale on HPC systems. However, in recent years, other distributed frameworks have entered the HPC domain, which do not rely on these properties, but require immense computing power and energy consumption. Providing state-of-the-art performance analysis tools from the HPC community to these new communities will be very beneficial for them. Examples of such frameworks are common machine learning/AI frameworks like PyTorch or TensorFlow; or Java-based frameworks such as Hadoop.

We are in the process of extending Score-P to provide its own distributed communication layer via a client/server model based on TCP. In contrast with the common MPI communication used in HPC application, where Score-P uses the same communication layer as the application, this communication layer will be used only by Score-P. This allows it to support future frameworks, independent of what new communication methods those frameworks may use. We foresee a wide range of applications for such models, including, but not limited to commercial cloud infrastructure, embedded development, and lastly Jupyter Notebooks.