Scalable Tools Workshop

What's New in Dyninst

Tim Haines, Jim Kupsch, Josef (Bolo) Burger

Abstract

Binary analysis and instrumentation is a key technology to support performance profiling, debugging, testing, software security, and auditing. Dyninst is an open source suite of libraries providing binary analysis, instrumentation and control capabilities across several hardware architectures with an architecture-independent abstraction. It supports both dynamic (runtime) and static (binary rewriting) instrumentation of a binary program. Dyninst is opportunistic in that it uses symbol and debug information when it is available, but can operate without it, even on stripped binaries. Dyninst's analysis capabilities produce a control- and data-flow analysis of the program, identifying functions, loops, and basic blocks in the code. Dyninst allows fine-grained program instrumentation and modification based on a high-level (control flow graph) abstraction of a program.

Dyninst is structured as a suite of toolkit libraries, providing architecture independent interfaces to features such as instruction decoding, control flow analysis, data flow analysis, code generation, code patching (splicing) and symbol table processing. On the dynamic side, it also includes process control and stack walking support.

Dyninst has been used as the foundation for products from companies like Cray and Red Hat, as the basis for tools from national labs and research groups, and as a key component in hundreds of academic research projects. It continues to have the due role of providing a foundation for new instrumentation and analysis research combined with support for key applications of binary analysis and instrumentation.

Since the last workshop in 2019, Dyninst has seen myriad improvements to functionality and code quality. Examples include support for instructions on the newly-minted Intel and AMD GPUs, parsing PIE binaries, and a wealth of correctness and usability enhancements discovered via user feedback, bug reports, our expanded test suites, and improved release testing. In this talk, we will provide a brief tutorial on how you can use Dyninst to take advantage of its capabilities and describe its new features and upcoming plans.

Preparing HPCToolkit for Performance Analysis for Exascale

Jonathon Anderson and Yumeng Liu

Abstract

Performance tools for emerging heterogeneous exascale platforms must address two principal challenges when analyzing execution measurements. First, measurement of large-scale executions may record mountains of performance data. Second, performance measurements for parallel programs are sparse in two ways: the set of metrics present for any context and the set of contexts present in different threads. For GPU-accelerated applications, an important source of sparsity is that none of the myriad of GPU metrics apply to any of the many CPU contexts.

To address these challenges, we developed a novel streaming aggregation approach to postmortem analysis that employs both shared and distributed memory parallelism to aggregate sparse performance measurements from every rank, thread, and GPU stream of an application, and attributes heterogeneous call path profiles and traces to source code. Analysis results are stored in a pair of sparse formats designed for efficient access to related data elements, supporting responsive interactive presentation and scalable data analytics. Using the same amount of resources, our approach analyzes large-scale performance measurements of GPU-accelerated applications over an order of magnitude faster than HPCToolkit and its sparse analysis results are as much as three orders of magnitude smaller than HPCToolkit's dense representation of metrics.

ONEVIEW: Guiding the User Through the Optimization 'Maze'

William Jalby

Abstract

We had a continuing effort in the past years has been to improve user guidance throughout the optimization process. This has been achieved through close interactions with code developers. In particular, we are working in the TREX project (focusing on bringing leading QMC Applications to the Exascale Level: https://trex-coe.eu/). This was an excellent opportunity to use our performance toolset but it also forced to change it in major ways to match more closely code developer needs. In this talk, we will present the new organization of our tool set: first, providing the user with a global estimation of code quality including not performance gains estimates but also transformation cost evaluations; second, displaying comparative views such as through different parallel settings, code versions, compilers, hardware; and third, providing user with multilevel analysis of parallel OMP loop constructs.

Improving Tool Support for Nested Parallel Regions with Introspection Consistency

Vladimir Indic

Abstract

The OpenMP 5 standard defines OMPT—an application programming interface for tools that includes a set of introspection routines. At any point in time, a sampling-based performance tool may invoke these introspection routines from a signal handler to inquire about the nesting of parallel regions and tasks. Unfortunately, the OpenMP 5 standard doesn’t precisely specify what one may observe with these routines when monitoring a program as it executes nested parallel regions. To address this shortcoming, we propose that the OpenMP standard require that an OpenMP implementation supports introspection consistency. In this talk, we definee introspection consistency, describes why tools need it, and explains a novel strategy for implementing it using wait-free coordination between an OpenMP implementation and its OMPT introspection routines. Additionally, we describe an implementation of this technique in the LLVM OpenMP runtime and evaluate the runtime overhead of supporting introspection consistency in LLVM OpenMP using microbenchmarks for nested parallel regions.

Possible Malleability Support in MPI 5: What Does it Mean for Tools?

Martin Schulz

Abstract

The Sessions working group of the MPI forum is currently working on extensions to the MPI standard that would enable applications to become malleable, i.e., be able to change the number of processes being used during the application's execution. This is driven, in part, by work in two large European projects focusing on a future European exascale software stack. While this would bring a new level of flexibility to write MPI applications, this would also make it harder for tools to capture, process and analyze information gathered from such applications. In this talk I will highlight the current proposals in the working group and will discuss possible consequences for tools.

LD_AUDIT and New Tools Interfaces

Ben Woodard

Abstract

The UNIX dynamic linker has had at least the specification an auditing interface since the heyday of Solaris. However, the quality of the implementation in both Solaris and glibc really did not live up to the design of the specification. Recently, a lot of work has gone into the LD_AUDIT interface in glibc to make it usable. This talk is an introduction to the LD_AUDIT interface, how it can be used to overcome weaknesses in the dynamic linker that impact HPC applications and tools. It concludes with some practical observations using LD_AUDIT as it now exists, and introduces the question about where it is not sufficient and where glibc tools interfaces need to go in the future.

A Software Microscope

Richard L. Sites

Abstract

Observation tools for understanding occasionally-slow performance in large-scale distributed transaction systems are not keeping up with the complexity of the environment. The same applies to large database systems, to real-time control systems in cars and airplanes, and to operating system design.

Extremely low-overhead tracing can reveal the true execution and non-execution (waiting) dynamics of such software, running in situ with live traffic. KUtrace is such a tool, based on small Linux kernel patches recording and timestamping every transition between kernel- and user-mode execution across all CPUs of a datacenter or vehicle computer. The resulting displays show exactly what each transaction is doing every nanosecond, and hence shows why unpredictable ones are slow, all with tracing overhead well under 1%. Recent additions to KUtrace also show interference between programs and show profiles within long execution stretches that have no transitions.

The net result is deep insight into the dynamics of complex software, leading to often-simple changes to improve performance.

Bio: Richard L. Sites wrote his first computer program in 1959 and has spent most of his career at the boundary between hardware and software, with a particular interest in CPU/software performance. His past work includes VAX microcode, DEC Alpha co-architect, and inventing the performance counters found in nearly all processors today. He has done low-overhead microcode and software tracing at DEC, Adobe, Google, and Tesla. Dr. Sites earned his PhD at Stanford in 1974; he holds 66 patents and is a member of the National Academy of Engineering.

Recent Advances in PAPI

Anthony Danalis

Abstract

PAPI has been a layer that provides a consistent interface to hardware performance counters for over two decades. This talk will outline the newest developments and the future directions of PAPI. GPUs have been critical to achieving high performance on modern systems, and multiple vendors have been producing solutions with different characteristics and APIs. Consequently, PAPI has been extending its support for GPUs across vendors to enable tools and application developers to access performance information on GPUs with minimum effort. Another focus of PAPI development has been the effort to categorize and understand the numerous counters that exist in modern hardware platforms. Lastly, we will discuss new features integrated in the Software Defined Events layer of PAPI and lessons learned from our efforts so far.

Noise-resilient Performance Measurement and Analysis of HPC Applications

Bernd Mohr

Abstract

Key to understanding and ultimately improving the performance of HPC applications is performance measurement. Unfortunately, many HPC systems expose their jobs to substantial amounts of interference (aka noise), leading to significant run-to-run variation. This makes performance measurements generally irreproducible, heavily complicating performance analysis and modelling. On noisy systems, performance analysts usually have to repeat performance measurements several times and then apply statistical analysis to capture trends. Firstly, this is expensive and secondly, extracting trends from a limited series of experiments is far from trivial, as the noise can follow quite irregular patterns.

In the German DFG project ExtraNoise we will develop noise-resilient performance measurement and analysis tools for HPC applications based on the well-known tools Score-P and Scalasca. The basic idea is to use logical timers, which are noise-insensitive, for measurement and analysis. We investigate logical timers on various levels: (1) function, communication, synchronization events, (2) basic blocks, (3) statements, and (4) noise-insensitive HW counters. We will present early results on the effectiveness of using logical timers for a noise-resilient performance measurement and analysis. We also will present NOIGENA, a noise generator application, we developed for testing our approach.

Noise-Resilient Performance Modeling of HPC Applications

Felix Wolf

Abstract

The key to understanding and ultimately improving the performance of HPC applications is performance measurement. Unfortunately, many HPC systems expose their jobs to substantial amounts of interference (aka noise), leading to significant run-to-run variation. This makes performance measurements generally irreproducible, heavily complicating performance analysis and modeling. On noisy systems, performance analysts usually have to repeat performance measurements several times and then apply statistical analysis to capture trends. Firstly, this is expensive and secondly, extracting trends from a limited series of experiments is far from trivial, as the noise can follow quite irregular patterns. In the DFG project ExtraNoise we will develop noise-resilient performance measurement, analysis, and modeling methods for HPC applications based on the well-known tools Score-P, Scalasca, and Extra-P.

This talk focuses on one of the project tasks – to empower the performance-modeling toolchain around Extra-P to deal with noisy performance data. The central idea is to introduce a stronger prior into the learning process. The prior will be derived from more or less noise-immune analyses, including both static and dynamic techniques, and support the model search for noise-affected performance metrics such as time, but also energy.

sys-sage: A Fresh View on the Topology of HPC Systems

Stepan Vanecek

Abstract

In order to use modern HPC systems efficiently, one has to be aware of the architectural details, in particular details of the hardware topology. hwloc has been used as a de-facto standard solution for determining the hardware topology of a system and for scheduling by runtime systems, such as the ones of OpenMP or MPI, or even by applications themselves. However, on ever-growing systems and with growing application complexity and increasingly more dynamic behavior, knowing the static hardware topology alone is not sufficient anymore.

I will present our approach, called sys-sage, and its library implementation that goes beyond hwloc's functionality to provide a more complete view of a node or a system. It can represent the dynamic state of the system and increases the degree of detail compared to static hardware topology views. The novelty of our approach lies in the ability to capture dynamic environments as well as systems' complexities, and in enabling greater flexibility in its usage.

EasyView: Bridging Software Developers with Dynamic Program Analysis

Xu Liu

Abstract

Profiling is well-known for its powerful capabilities of identifying performance inefficiencies in complex code bases. Existing profilers suffer from several usability issues. First, the profilers are disjoint from the coding environments such as IDEs; frequently switching focus between them significantly complicates the entire cycle of software development. Second, mastering multiple tools to interpret their analysis results requires substantial efforts; moreover, many tools have their own design of graphic user interfaces (GUI) for data presentation, which deepens the learning curves. Third, most existing tools expose few interfaces to support user-defined analysis, which makes the tools less customizable to fulfill the diverse user demands.

We develop EasyView, a general solution to integrate the interpretation and visualization of various profiling results in the coding environments, which bridges software developers with profilers to provide easy and intuitive dynamic analysis during the code development cycle.

Autotuning MPI/OpenMP ECP Proxy Applications at Large Scales

Xingfu Wu

Abstract

As we enter the exascale computing era, efficiently utilizing power and optimizing the performance of scientific applications under power and energy constraints has become critical and challenging. In this talk, we propose a low-overhead, scalable autotuning framework to autotune both performance and energy for various hybrid MPI/OpenMP applications at large scales. We use the proposed autotuning framework to tune four MPI/OpenMP ECP proxy applications - XSBench, AMG, SWFFT, and SW4lite. Our approach uses Bayesian optimization with a Random Forest surrogate model to effectively search parameter spaces with up to 6 million different configurations on two large-scale production systems, the Cray XC40 Theta at Argonne National Laboratory and the IBM Power9 heterogeneous system Summit at Oak Ridge National Laboratory. The experimental results show that our autotuning framework at large scales has low overhead and achieves good scalability.