Scalable Tools Workshop

What's New in Dyninst

Dyninst Team: James Kupsch, Ronak Chauhan, Angus He

Abstract

Binary analysis and instrumentation is a key technology to support performance profiling, debugging, testing, software security, and auditing. Dyninst is an open source suite of libraries providing binary analysis, instrumentation and control capabilities across several hardware architectures with an architecture-independent abstraction. It supports both dynamic (runtime) and static (binary rewriting) instrumentation of a binary program. Dyninst is opportunistic in that it uses symbol and debug information when it is available, but can operate without it, even on stripped binaries. Dyninst's analysis capabilities produce a control- and data-flow analysis of the program, identifying functions, loops, and basic blocks in the code. Dyninst allows fine-grained program instrumentation and modification based on a high-level (control flow graph) abstraction of a program.

Dyninst is structured as a suite of toolkit libraries, providing architecture independent interfaces to features such as instruction decoding, control flow analysis, data flow analysis, code generation, code patching (splicing) and symbol table processing. On the dynamic side, it also includes process control and stack walking support.

Dyninst has been used as the foundation for products from companies like AMD, Cray, and Red Hat, as the basis for tools from national labs and research groups, and as a key component in hundreds of academic research projects. It continues to have the dual role of providing a foundation for new instrumentation and analysis research combined with support for key applications of binary analysis and instrumentation.

Since the last workshop, Dyninst has seen myriad improvements to functionality and code quality. Examples include AMD MI300 GPU support; AMD GPU indirect branch support; improved x86 instruction parsing, mnemonics, missing instructions, and properties (syscall, interrupt, NOP); improved instruction formatting; improved DWARF type and CU parsing; improved parsing line maps; new compiler and glibc support; and CI improvements.

In this talk, we will review the features and structure of Dyninst, summarize the new developments, and present some practical examples of how you can use it.

HPCToolkit: New Support for GPU Measurement

John Mellor-Crummey and Yuning Xia

Abstract

There have been significant developments in performance monitoring support for GPUs from AMD and Intel over the last year. In this talk, we will describe a prototype implementation of HPCToolkit atop AMD’s emerging rocprofiler-sdk and new support for PC sampling for Intel GPUs.

HPC Application Performance Monitoring and Analysis with LDMS

Jennifer Green and Ann Gentile

Abstract

The Lightweight Distributed Metric Service (LDMS) is an open-source, state-of-the-art monitoring system widely used on large-scale systems across DOE labs. It provides scalable, “always-on”, time-correlated collection of both system and application resource utilization and performance information. It collects data at frequencies enabling resolution of features of interest, while still having minimal negative performance impact. This talk will present an overview of LDMS and its design features. It will include a deep-dive into a new design for scalable event transmission that is foundational to enabling runtime, analysis-driven feedback for low-latency response to performance degrading circumstances. We further present production use cases showing how LDMS has been, and is being, utilized to provide insights for both application users/developers and system operations staff.

ZeroSum: User Space Monitoring of Resource Utilization and Contention on Heterogeneous HPC Systems

Kevin Huck

Abstract

Heterogeneous High Performance Computing (HPC) systems are highly specialized, complex, powerful, and expensive systems. Efficient utilization of these systems requires monitoring tools to confirm that users have configured their jobs, workflows, and applications correctly to consume the limited allocations they have been awarded. Historically system monitoring tools are designed for - and only available to - system administrators and facilities personnel to ensure that the system is healthy, utilized, and operating within acceptable parameters. However, there is a demand for user space monitoring capabilities to address the configuration validation and optimization problem. In this paper, we describe a prototype tool, ZeroSum, designed to provide user space monitoring of application processes, lightweight processes (threads), and hardware resources on heterogeneous, distributed HPC systems. ZeroSum is designed to be used either as a limited-use porting tool or as an always-on monitoring library.

GPU compute kernel instrumentation and performance analysis for HPC

Sébastien Darche

Abstract

The DORSAL group at Polytechnique Montréal, in collaboration with Ericsson, Ciena, AMD and EfficiOS, is conducting research on tracing, profiling, debugging and performance analysis tools, for large Cloud and HPC distributed applications.

We present in this talk ongoing work in our group on the HPC tracing ecosystem and the scalability of trace processing in the TraceCompass trace analysis framework. We will focus on low-overhead tracing techniques for GPU compute kernels, which enable the kernel developers to gather traces directly on the device. Our method features a reduced instrumentation noise compared to similar approaches, thus allowing for precise time measurements. We also present current work on buffering techniques on the device which would allow for continuous tracing on the GPU.

Assessing CPU Code Quality

Emmanuel Oseret, Cedric Valensi, William Jalby

Abstract

Code quality is essential for getting high performance: for various reasons (poor performance models, lack of adequate transformations, …) compilers are often producing suboptimal codes, which can significantly hurt performance.

MAQAO is a performance analysis framework offering features designed for assessing CPU (X86 and ARM) code quality, detecting potential compiler “mistakes” and suggesting workarounds (inserting compiler flags, rewriting loops, ...).

In our presentation, we will focus on some of these features and in particular on how they can be used to compare compilers and import optimizations between them. These MAQAO capabilities will be illustrated through real code examples.

Register Availability Analysis in Support of Instrumentation of GPU Kernels

Hsuan-Heng Wu

Abstract

Binary code instrumentation has a long history for being used for performance analysis, debugging, software reliability and security purposes on CPU binaries. In contrast to CPU binary instrumentation, instrumenting GPU binaries poses some unique challenges: 1. GPU kernels do not necessarily have access to a general purpose stack or heap, which is commonly used in conventional CPU instrumentation. 2. Highly optimized GPU kernels can use up all architecturally available registers, and if these kernels have no stack available to support register spilling, we end up with little or none registers available for instrumentation.

In this talk, we attempt to address the two challenges at the same time by answering the following question: What is the minimal register requirement required to enable access to a general-purpose stack and heap, which in turn support the "conventional" instrumentation. Based on whether the kernel is compiled with scratch(stack), and based on the number of registers available for instrumentation, we come up with different instrumentation strategies to support conventional instrumentation with increasing overheads.

We inspect the 738 kernels in the MIOpen library across the MI100, MI200 and MI300 GPU architectures, categorize them based on the instrumentation strategies (with default allocation or by increasing the register allocation when possible), and show that conventional instrumentation can be supported for more than 97% of the kernels in this vendor-provided hardware-optimized machine learning library.

GPUscout: Locating Data Movement-related Bottlenecks on (for now NVidia) GPUs

Stepan Vanacek

Abstract

GPUs pose an attractive opportunity for delivering high-performance applications. However, GPU codes are often limited due to memory contention, resulting in overall performance degradation. Since GPU scheduling is transparent to the user, and GPU memory architectures are very complex compared to ones on CPUs, finding these bottlenecks can be a very cumbersome process.

GPUscout is a novel method of systematically detecting the root cause of frequent memory performance bottlenecks on NVIDIA GPUs. It connects three approaches to analyzing performance - static CUDA SASS code analysis, sampling warp stalls, and kernel performance metrics. Connecting these approaches, GPUscout can identify the problem, locate the code segment where it originates, and assess its importance. Therefore, the user is informed both about the type of the issue and where exactly it is located in the source code. Combined with additional selected kernel-wide metrics, it provides the necessary information to evaluate the severity of the potential bottleneck, and in the next step, the effect that the implemented fix has had.

AMD ROCm Tools Update: The New RocProfiler-SDK and Updates to Debugger, OMPT, and Binary Instrumentation Support

Benjamin Welton

Abstract

Approximately one year ago, we started soliciting feedback on how we could better improve the performance tool experience in the HPC and AI workload space. That feedback resulted in a ground up re-write of our profiler and tracing libraries to resolve the common pain points tool developers faced when using these libraries as well as improvements in both performance and data quality. Rocprofiler-sdk, the upcoming replacement for tools like rocprofiler and roctracer, corrects initialization ordering issues and allows multiple APIs/tools requiring simultaneous use of the library. Rocprofiler-SDK also adds concurrency support, buffer management, while improving the performance or profiling on both tracing and counter collection workloads. We will be giving an overview of Rocprofiler-SDK and discussing the performance changes you can expect when the beta is released in ROCM 6.3. We will also be giving updates to the debugger, OMPT, and binary instrumentation support.

Programmatic Analysis of Large-Scale Performance Data

Dragana Grbic

Abstract

HPCToolkit typically measures entire program executions, capturing fine-grained measurement data, which may include instruction-level samples on CPUs and GPUs. When analyzing applications running at exascale, HPCToolkit’s performance databases can become huge, making manual inspection difficult and time-consuming. We explored existing tools including Hatchet and Thicket for programmatic analysis of performance data to automate this process. However, HPCToolkit’s calling context trees are difficult to interpret and visualize using these tools because they often contain many nodes that represent contexts where little cost has been incurred. Moreover, importing HPCToolkit databases for multiple application runs into Thicket was slow as unifying calling context trees for all runs was costly for large trees. To accelerate matching of calling context trees, we employ filtering to reduce the calling context trees to the most valuable sections for analysis. Additionally, we developed a new API to efficiently extract slices of HPCToolkit’s performance data specified by query expressions. We plan to integrate use of the new API into Hatchet and Thicket to provide users with efficient techniques to programmatically analyze large-scale performance data.

Thicket: Growth of the Heterogeneous Performance Experiment Forest

Dewi Yokelson

Abstract

Comparison and analysis of large-scale application codes across multiple runs can enable deeper understanding of the optimal performance configuration. Thicket is an open-source Python toolkit that provides support for this type of comparative analysis across diverse architectures and tools. This talk provides background on Thicket as well as sharing recent updates and upcoming features. First, we discuss how Thicket works, and demonstrate its capabilities in two case studies. Then we discuss new functionality — including the support of heterogeneous (GPU) computing resources, extended statistical and query language functionality, and time series performance analysis and visualization.

Spack-Based Production Programming Environments on Cray Shasta

Francine Lapid

Abstract

The High Performance Computing (HPC) Programming and Runtime Environments Team (PRETeam) at Los Alamos National Laboratory (LANL) supports mission-critical science by providing and maintaining user software. As the current fleet of supercomputers at LANL transitions to being exclusively Cray Shastas, providing software poses new challenges. Particularly, making the software stack compatible with the Cray Programming Environment often requires additional steps. To improve the legacy process of installing individual applications via their own custom scripts, the PRETeam has opted to take advantage of Spack’s many features including its vast library of package recipes, automatic modulefile generation, buildcache production, etc. In this presentation we detail how the PRETeam is implementing solutions, resolving challenges encountered, the benefits to the PRETeam and the users, and future improvements to the workflow.