Scalable Tools Workshop 2026

What's New in Dyninst

James Kupsch and Hsuan-Heng Wu

Abstract

Binary analysis and instrumentation is a key technology to support performance profiling, debugging, testing, software security, and auditing. Dyninst is an open source suite of libraries providing binary analysis, instrumentation and control capabilities across several hardware architectures with an architecture-independent abstraction. It supports both dynamic (runtime) and static (binary rewriting) instrumentation of a binary program. Dyninst is opportunistic in that it uses symbol and debug information when it is available, but can operate without it, even on stripped binaries. Dyninst's analysis capabilities produce a control- and data-flow analysis of the program, identifying functions, loops, and basic blocks in the code. Dyninst allows fine-grained program instrumentation and modification based on a high-level (control flow graph) abstraction of a program.

Dyninst is structured as a suite of toolkit libraries, providing architecture independent interfaces to features such as instruction decoding, control flow analysis, data flow analysis, code generation, code patching (splicing) and symbol table processing. On the dynamic side, it also includes process control and stack walking support.

Dyninst has been used as the foundation for products from companies like AMD, Cray, and Red Hat, as the basis for tools from national labs and research groups, and as a key component in hundreds of academic research projects. It continues to have the dual role of providing a foundation for new instrumentation and analysis research combined with support for key applications of binary analysis and instrumentation.

Since the last workshop, Dyninst has seen myriad improvements to functionality and code quality. Examples include more accurate CFGs, significant performance improvements for large binaries, fixes to several instruction's semantics, refactoring to support cross architecture instrumentation, and initial support for AMD GPU and RISC-V architectures.

As always, there are updates to testing and a wealth of correctness, performance, and usability enhancements discovered via user feedback, bug reports, and our testing process.

In this talk, we will review the features and structure of Dyninst, summarize the new developments, and present some practical examples of how you can use it.

Top-Down Microarchitecture Analysis in HPCToolkit

Laksono Adhianto

Abstract

The Top-Down Microarchitecture Analysis (TMA) methodology provides a systematic framework for identifying CPU performance bottlenecks by hierarchically categorizing pipeline slots into four root causes: Frontend Bound, Backend Bound, Bad Speculation, and Retiring. Originally designed for coarse-grain measurement at the application or function level, TMA has seen broad adoption across modern Intel microarchitectures, including Skylake, Ice Lake, and Sapphire Rapids, as well as analogous implementations on AMD (RDTSC-based pipeline analysis) and Arm (Neoverse TMA). Its strength lies in its structured, metric-driven diagnosis; its principal limitation is the sheer number of hardware performance counters it demands, making full-tree evaluation costly and, in many configurations, impractical within a single profiling run.

HPCToolkit is a sampling-based performance analysis tool designed for fine-grain, instruction-level measurement on large-scale HPC systems. By attributing performance metrics to individual instructions within full calling-context trees, HPCToolkit exposes bottlenecks invisible to function- or application-level profilers. Integrating TMA into this paradigm, however, introduces fundamental tensions. TMA's slot-based metrics are defined over aggregate pipeline behavior, whereas HPCToolkit's sampling model captures point-in-time hardware state at the instruction level — a mismatch in both granularity and temporal semantics.

This talk describes our ongoing work to reconcile these two models. We present an implementation of TMA for Intel Sapphire Rapids in HPCToolkit that uses hardware counter sampling to collect per-instruction observations and then attributes the resulting metrics upward to loops, functions, and whole programs. To manage counter multiplexing overhead, we expose a selective subtree interface — allowing users to target a specific branch of the TMA hierarchy — so that the required counter budget remains within the available PMU's constraints.

What's New in Caliper

David Boehm

Abstract

Caliper is LLNL's instrumentation and profiling library that integrates performance analysis capabilities directly into HPC codes, supporting a wide range of performance engineering use cases, from always-on runtime profiling to detailed performance debugging and large-scale run-to-run performance comparison studies. This talk gives a short recap of Caliper's capabilities, and provides an overview of many new (and some not-so-new) features in Caliper and its surrounding software ecosystem, including memory allocation profiling, MPI communication statistics, Python support, updates to AMD GPU support, and run metadata capture. We cover these features from a tool developer's perspective and discuss issues we found, open questions, and lessons learned.

Experiences with Top-down Performance Analysis of GPUs

John Mellor-Crummey

Abstract

In 2025, we extended the HPCToolkit performance tools with support for performance analysis within GPU kernels using PC sampling on AMD and Intel GPUs, complementing existing support for NVIDIA GPUs. Hardware support for PC sampling on AMD's MI300A is particularly informative, enabling HPCToolkit to measure and attribute cycles, stalls, and stall reasons, in addition to other metrics such as utilization of wavefronts and individual GPU pipelines. We have been using this information to build top-down performance models for GPUs. In this talk, I will report about our top-down models for AMD, Intel and NVIDIA GPUs, some questions that arose when using them to gain insight into issues degrading application performance, some resulting adjustments to our approach, and feedback for vendors.

PAPI: Tackling AMD and NVIDIA’s Emerging Architectures

Treece Burgess

Abstract

As AMD and NVIDIA introduce increasingly complex architectures, state-of-the-art profiling tools are needed to analyze them; as a result, PAPI is challenged to stay at the forefront. PAPI has met this challenge for AMD through the amd_smi and rocp_sdk components, and for NVIDIA through the cuda and nvml components. In this talk, we showcase the amd_smi, rocp_sdk, and cuda components by first exploring architecture and API support, next discussing challenges and experiences, and lastly outlining future actions.

PAPI Support for AI Architectures, Code Annotation, and PC Sampling

Daniel Barry

Abstract

The rise of specialized hardware for AI workloads has necessitated commensurate support in PAPI. Additionally, applications’ use of code annotation libraries, such as ROCTX, needs to be acknowledged during the event monitoring process. Furthermore, Program Counter (PC) sampling—available for various AMD GPUs and APUs—allows for richer profiling when used in concert with conventional event monitoring. In this presentation, we showcase the recent efforts in PAPI to support AI architectures, interoperate with code annotation, and integrate PC sampling.

Scalable Launch on El Capitan with Spindle

Matt Legendre

Abstract

Job launch on the world's largest supercomputer is fast and scalable thanks to Spindle, which handles scalable library loading for every job that runs on El Capitan. This talk will discuss the new capabilities and scalability improvements needed in Spindle for El Capitan, and it will discuss the adaptations needed by the increasingly AI-centric workloads found on the system. It will highlight some of the remaining complications that need continued R&D, such as integration/combability with other tools and the limitations around using the LD_AUDIT interface. Finally, we will propose paths forward on these issues for further discussion.

STICM: Smart Tools and Interface for Complex Memory

Terry Jones

Abstract

Smart Tools and Interfaces for Complex Memory (STICM) is joining the DOE STEP project as a new tool based on the SICM (Simplified Interface to Complex Memory) platform. This talk will describe our previous work with SICM and our planned future directions and stewardship as part of the STEP portfolio.

What is Going on with Debugging Info? DWARF6 and Support for GPUs and Vectorized Code

Ben Woodard

Abstract

Work continues on DWARF6 with the biggest new feature in this forthcoming version of the DWARF standard being standardized debug information for vectorized code and GPUs. With all major GPU vendors now participating in this endeavor there has been lively discussion and considerable progress has been made. Many new features have been accepted while others are still being edited or revised. This talk covers what to expect in the forthcoming DWARF6 standard with a particular focus on the facilities that are being added to support GPUs and vectorized code.

Performance Analysis Tools and SDK for Intel® GPUs: Today and Tomorrow

Zhiqiang Ma

Abstract

Performance analysis tools are essential for understanding application behavior and identifying bottlenecks in AI and HPC workloads. This talk focuses on the open-source PTI SDK and unitrace tool and discusses future directions in performance analysis for Intel® GPUs.

The PTI SDK provides APIs that enable developers to build custom profiling tools or add profiling capabilities to existing tools, frameworks, such as PyTorch, and applications. The unitrace is a lightweight and powerful tool for tracing host and device activities, and profiling device hardware performance metrics. We will use practical examples to demonstrate how to use these tools to analyze AI and HPC applications. We will also discuss emerging approaches using AI to process profiling data, identify performance bottlenecks, and accelerate optimization workflows.

New Developments in the ROCm Tools Ecosystem

Anthony Danalis

Abstract

Effective performance optimization on AMD GPUs depends on having the right tools - and knowing how to use them. This talk gives an overview of the ROCm performance analysis ecosystem: what each tool does, where it fits in an optimization workflow, and especially what's new.

We will cover recent developments across the stack, including new features in Rocprofiler-SDK, the foundational library that powers internal and external tools, and highlight new interfaces available for third-party tool authors who want to build on top of it.

We will also introduce an experimental effort to make profiling more accessible to both developers and AI agents: a ROCm profiling skill and tool library designed to help collect, interpret, and act on profiling evidence within existing workflows — a step toward treating performance analysis as something that can be reasoned about, not just run.

Network and Storage Profiling

Holly Wilper

Abstract

Performance tooling at scale is an ongoing issue in the HPC and Cloud computing environment. NVIDIA has extended its suite of tools a great deal over the last few years to better support the use cases in this environment.

We’ll start with a quick overview of NVIDIA DevTools and new features. Then we will turn to a deeper dive into network hardware support that has been added to Nsight Systems, including NICS, Switches, fabrics, and storage devices. We’ll touch on improvements in the NVTX annotation library to support arbitrary structured data and how that leads to the plugin methodology that is enabling multiple hardware venders to expose their data through Nsight Systems. Finally, we’ll touch a little bit on NVIDIA Tools support for AI agents in the modern computing world.