Sparkles

Appearance

Linux perf_events ​

The survey's reference model: one syscall — perf_event_open(2) — is the single acquisition hub for both counting and sampling, and four independent libraries decode what it produces without ever touching the PMU.

FieldValue
Subsystemperf_events (kernel kernel/events/) + the tools/perf userspace consumer
Entry pointperf_event_open(2) — returns one fd per event; fds compose into groups
ABIstruct perf_event_attr (uAPI include/uapi/linux/perf_event.h) in, records + counter reads out
Kernel readv7.1-rc6, linux@e43ffb69e043
RoleHub-and-decoders: the syscall acquires; libdwfl/libtraceevent/libnuma interpret
Decoderscode space (libdwfl) · event space (libtraceevent) · topology (libnuma)
Verification[hw-verified: x86_64-linux] — Linux 6.18.26, AMD Ryzen 9 7940HX (Zen 4), LDC 1.41

NOTE

All three hardware experiments on this page were recorded on Linux 6.18.26, an AMD Ryzen 9 7940HX (Zen 4, 6 general-purpose core PMCs), with /proc/sys/kernel/perf_event_paranoid = -1, built with LDC 1.41 over druntime's core.sys.linux.perf_event. Each is a runnable probe under examples/ that CI compiles and runs.

Overview ​

What it acquires ​

Every other page in this survey describes a variation on, or a decoder for, the machinery here. Linux is the survey's reference model for a reason: it is the one platform where a single, uniform, unprivileged-capable syscall covers the whole acquisition surface. perf_event_open(2) opens one event — described by a struct perf_event_attr — and returns a file descriptor. Whether that event counts (accumulate one number, read it with read(2)) or samples (write a record stream into an mmap'd ring on every counter overflow) is a property of the same attr, not a different API. Windows splits this across ETW, the HAL, and ReadThreadProfilingData; macOS across kpc and kperf; Linux unifies it.

The perf_event_attr ABI is wide but flat: a major type (PERF_TYPE_HARDWARE, PERF_TYPE_TRACEPOINT, …), a config number selecting the event within that type, and a bitfield block of modifiers (disabled, exclude_kernel, exclude_hv, precise_ip, sample_type, …) (perf_event.h:393 for the struct, :418 for the modifier block). linux@e43ffb69e043 include/uapi/linux/perf_event.h. [source-verified]

The hub-and-decoders model ​

The load-bearing architectural fact — the one that shapes every other page — is that none of the four decoder libraries is required to acquire data; each is required only to interpret it. perf_event_open and read/mmap give you raw numbers, raw instruction pointers, and raw record blobs. Turning those into meaning is delegated:

DecoderDecodesFromOwns concern
libelf/libdw/libdwflcode spaceIP → module → symbol → line + DWARF-CFI unwind4
libtraceeventevent spacea tracefs format schema → typed field extraction5
libnumatopologysysfs /sys/devices/system/node + mempolicy syscalls6
libpfm4 etc.naminghuman event name → attr.{type, config}7

A group is scheduled onto the PMU as one atomic unit — the property that makes a valid IPC possible (counting cycles and instructions over the same window). If the whole group cannot fit on the hardware it is rolled back and not scheduled at all, verbatim from the kernel (kernel/events/core.c:2886):

/* Groups can be scheduled in as one unit only, so undo any partial group before returning: */

That all-or-nothing contract is why a benchmarking harness sizes its group to the physical counter budget rather than hoping the kernel partial-counts — it never will. [source-verified]

How it works ​

The lifecycle of an event is four phases: open (perf_event_open with a filled attr), arm (PERF_EVENT_IOC_RESET + PERF_EVENT_IOC_ENABLE via ioctl, :576), run (the workload executes), harvest (read(2) for counting; drain the mmap ring for sampling). Grouping and enable/disable can address a whole group at once with the PERF_IOC_FLAG_GROUP ioctl flag: enabling the leader enables every sibling.

The kernel side is a per-CPU/per-task context of scheduled events. When more events are enabled than there are counters, a per-CPU hrtimer (perf_rotate_context, core.c:4587) time-slices them, and the kernel accounts, per event, how long it was enabled versus actually running on hardware (__perf_update_times, core.c:744). Those two timestamps are the entire basis of multiplexing scaling, below. [source-verified]

The seven concerns ​

This survey scores every backend against the same seven concerns. Linux is where five of them are acquired and two (3, then the decode halves of 4–7) are deferred to a decoder. The concern order is fixed across the tree.

Scalar counting: groups, PERF_FORMAT_GROUP, and multiplexing ​

Concern 1 — acquired here. A counting event is opened with attr.disabled = 1 on the leader; sibling events pass the leader's fd as group_fd, which joins the group (perf_event.h:393). The leader carries the read format. With PERF_FORMAT_GROUP set, a single read(2) on the leader returns the whole group atomically as {nr, time_enabled, time_running, value[nr]} (read-format enum :365; producer perf_read_group, core.c:6154, with nr = 1 + leader->nr_siblings). Counting is cheap: no interrupts, no buffers, one register read per counter, near-zero perturbation — the natural mode for a per-iteration benchmark harness. [source-verified]

Multiplexing. When enabled events exceed the counter budget, the reader must scale each raw count by time_enabled / time_running to estimate the full-window value. The exact formula is documented in the uAPI header itself (perf_event.h:685):

text
/* include/uapi/linux/perf_event.h:685 */
 *   quot = count / running;
 *   rem  = count % running;
 *   count = quot * enabled + (rem * enabled) / running;

Scaled counts are estimates: a sub-millisecond running slice produces visible error, and an event that got zero PMU time (running == 0) reads as perf's <not counted> — a zero read is not a measurement. A harness avoids the whole problem by keeping its group inside the physical counter budget — the reason the sparkles baseline drops its LLC pair when calibration detects rotation. [source-verified]

Experiment A — counting-group.d — reads a fitting cycles+instructions group (exact IPC, scale == 1), then deliberately oversubscribes 10 instruction counters onto 6 Zen-4 PMCs to force multiplexing:

text
== Demo 1: fitting group (kernel+user) ==
  nr=2  time_enabled=3020595 ns  time_running=3020595 ns  scale=1.0000
  cycles=15099126  instructions=60049109  IPC=3.977
== Demo 2: 10 instruction counters, 6 general-purpose PMCs ==
  ev   raw_running_count      enabled_ns    running_ns   scale   estimate
   0             13444336        3337469        762700    4.38   58830542
   3             60058404        3332911       3332911    1.00   60058404   ← fit, exact
   7             10661419        3320297        582933    5.70   60725808
   8                    0        3316961             0       —   <not scheduled>
  multiplexing observed: yes

The scaled estimates cluster at the true ~60.05M retired instructions; two events got zero PMU time (perf's <not counted> state); the extreme 5.70× scale from a 0.58 ms slice is visibly the noisiest recovery — a caution for the harness's LLC-pair-drop heuristic and any pinned, oversubscribed group. [hw-verified: x86_64-linux]

rdpmc self-monitoring and the mmap-page seqlock ​

The lowest-overhead counter read skips the syscall entirely. The mmap'd perf_event_mmap_page (page 0 of the ring mapping, perf_event.h:596) publishes the counter index, offset, pmc_width, and a cap_user_rdpmc capability bit (:646/:663). The self-monitoring read is a userspace seqlock loop that reads the counter with the rdpmc instruction, verbatim from the uAPI (perf_event.h:608):

text
/* include/uapi/linux/perf_event.h:608 */
 *   do {
 *     seq = pc->lock;
 *     barrier()
 *     ...
 *     index = pc->index;
 *     count = pc->offset;
 *     if (pc->cap_user_rdpmc && index) {
 *       width = pc->pmc_width;
 *       pmc = rdpmc(index - 1);
 *     }
 *     barrier();
 *   } while (pc->lock != seq);

The kernel republishes the page under an incrementing lock seqcount in perf_event_update_userpage (core.c:6825, ++userpg->lock + barrier()). One caveat for the per-ISA pages that follow: cap_user_rdpmc and pmc_width are set by arch code, not the generic core — the __weak arch_perf_update_userpage stub at core.c:6815 is a no-op, and x86 fills the fields in arch/x86/events/core.c. The ARM arm_pmuv3 userpage path is its own story (see arm.md). [source-verified]

Overflow sampling: the ring buffer, PERF_RECORD_MMAP2, and IP symbolization ​

Concern 2 — acquired here. Sampling arms attr.sample_period (or attr.sample_freq with attr.freq = 1) plus a sample_type bitmask selecting which fields each record carries (PERF_SAMPLE_IP = 1<<0 … STACK_USER = 1<<13 … DATA_SRC = 1<<15, :142). On each overflow the PMI handler writes a PERF_RECORD_SAMPLE (record type 9, :1059) into the mmap ring: page 0 is the control perf_event_mmap_page (data_head/data_tail, :749/:750); pages 1.. are the data area. The sample's field order (IP, TID, TIME, …, REGS_USER, STACK_USER, …) is fixed by the sample_type bit order and documented at the record comment :1061. [source-verified]

Consumer ordering. The reader-side contract is a seqcount: acquire-load data_head, process [data_tail, data_head) wrapping modulo the data size, then release-store the new data_tail back. Experiment B parses this directly. [hw-verified: x86_64-linux]

The MMAP2 / stale-binary hazard. The single biggest surprise for anyone writing a self-profiler: PERF_RECORD_MMAP2 (record type 10, :1091) is emitted only for executable mappings created while the event is enabled. The kernel does not re-emit already-mapped code — your own binary, libc, everything that was mapped before you enabled sampling produces zero MMAP2 records. This is why the perf tool synthesizes MMAP2 for pre-existing regions from /proc/PID/maps at start, and why libdwfl's dwfl_linux_proc_report reads that same file. For a harness this is a real build-id/stale-binary hazard: symbolization must lean on /proc/self/maps or synthesize MMAP2, and must validate build-ids or risk silently symbolizing against a rebuilt binary. Perf's build-id path validates and rejects on mismatch (build-id.c:863, dso__build_id_mismatch). [hw-verified: x86_64-linux] — the probe captured 0 MMAP2 records until a mid-window mmap(PROT_EXEC) of /proc/self/exe forced one.

Experiment B — sampling-symbolize.d — samples cycles at ~4 kHz, drains SAMPLE + MMAP2 from the ring, and hands the IPs to libdwfl:

text
captured: 1 PERF_RECORD_MMAP2 mappings, 1996 IP samples
top self-symbols (dwfl: name — samples — file:line):
  …sumSquares…   1852   sampling-symbolize.d:137
  …mixHash…       130   sampling-symbolize.d:129
PERF_RECORD_MMAP2 captured during the window: 1 record(s)
  [0x798b80c68000, 0x798b80c69000) pgoff=0x0 prot=0x5 …/build/cpu_pmu_sampling_symbolize
model check: hottest symbol …sumSquares… (IP 0x55c5…) was symbolized by libdwfl,
and a captured MMAP2 names the same image.

The IPs land purely on the acquisition side here; the address → symbol → line decode is elfutils' job — see elfutils.md § "Address → module → symbol → line". [hw-verified: x86_64-linux]

Precise sampling and data-source attribution ​

Concern 3 — deferred to precise-sampling.md.perf_event_open acquires the precise-sampling request and its payload — the attr carries precise_ip (0–3) and the sample_type bits PERF_SAMPLE_ADDR, PERF_SAMPLE_DATA_SRC, PERF_SAMPLE_PHYS_ADDR, PERF_SAMPLE_WEIGHT — but the semantics (Intel PEBS, AMD IBS, ARM SPE, the perf_mem_data_src union, skid removal, and node classification) are one whole page of their own. This concern is covered end-to-end in precise-sampling.md; the data-source attribution payload it decodes is captured through this same syscall.

Code-space decode: the record → address model and stack capture ​

Concern 4 — address model here, decode → elfutils.md. The kernel hands you records; turning them into a symbol requires an address-space model (which file is mapped where) and a debug-info decoder. Linux acquires the first and defers the second to libdwfl.

The perf consumer pipeline (the address model). Perf's own record → symbol path is the reference wiring: MMAP2 → machine__process_mmap2_event builds a struct dso_id (the build-id if PERF_RECORD_MISC_MMAP_BUILD_ID is present, else maj/min/ino) + map__new + thread__insert_map (machine.c:1728); then a sampled IP resolves via thread__find_map (a maps__find bsearch, maps.c:1122) → map__map_ip rebases the runtime VA to a file offset (map.h:107) → map__find_symbol → dso__find_symbol (rbtree containment, symbol.c:412). [source-verified]

Stack unwinding: STACK_USER + REGS_USER and DWARF CFI ​

For a call-graph profile on a frame-pointer-less build, the frame-pointer chain-walk is impossible, so the sample additionally carries the interrupted thread's register file (PERF_SAMPLE_REGS_USER) and a copied slab of its user stack (PERF_SAMPLE_STACK_USER) — attr.sample_regs_user is the register mask, attr.sample_stack_user the slab size. The DWARF-CFI replay that turns those into a backtrace is again elfutils' — see elfutils.md § "DWARF-CFI stack unwinding". This page owns only the capture.

Experiment C — unwind-stack-user.d — samples with REGS_USER+STACK_USER on a --frame-pointer=none build and reconstructs a 5-frame backtrace offline from .eh_frame/.debug_frame CFI:

text
captured: 125 samples with REGS_USER+STACK_USER (0 had ABI_NONE)
chosen sample: leaf IP 0x… (…level3…+0x50)  regs ABI=2  RIP=0x… RSP=0x… captured stack=2776 bytes
DWARF-CFI backtrace (5 frames, frame pointers OMITTED …):
  #0 …level3…+0x50   #1 …level2…+0x12   #2 …level1…+0x12   #3 …workload…+0x71   #4 …run…+0x29A

This is byte-for-byte the wiring perf uses in tools/perf/util/unwind-libdw.c. The perf → DWARF x86-64 register permutation lives in the probe; ARM and RISC-V need their own maps (see arm.md / riscv.md). [hw-verified: x86_64-linux]

Event-space and tracing: PERF_TYPE_TRACEPOINT + PERF_SAMPLE_RAW ​

Concern 5 — raw blob here, decode → libtraceevent.md. Beyond hardware counters, perf_event_open counts and samples kernel software-event schemas. Opening an event with attr.type = PERF_TYPE_TRACEPOINT and attr.config set to a tracepoint's numeric id makes that tracepoint countable; adding PERF_SAMPLE_RAW to sample_type attaches the tracepoint's raw record blob to each sample. The blob is typed only by the tracepoint's tracefs format schema, which libtraceevent parses — see libtraceevent.md. Two boundaries matter here: the numeric id comes from reading .../events/<sys>/<event>/id under tracefs (whose files are root-only on hardened boxes — the gate the sparkles --syscalls layer hits on this machine), and libtraceevent decodes the blob but does not read the tracefs files (libtracefs does). [source-verified]

NUMA and topology ​

Concern 6 — n/a to the syscall → libnuma.md. The perf_event_open syscall has no topology concern — it neither enumerates nodes nor attributes a sampled data address to a node. A sample can carry the data virtual/physical address (PERF_SAMPLE_ADDR/PERF_SAMPLE_PHYS_ADDR, concern 3), but answering "which node backs this page?" is a separate decoder: sysfs /sys/devices/system/node topology plus the mempolicy syscalls, wrapped (mostly) by libnuma. See libnuma.md — including the missing VA→node helper that precise-sampling.md must open-code.

Event naming and encoding ​

Concern 7 — opaque numbers here → event-naming.md. On this page attr.type and attr.config are opaque numbers. The syscall takes PERF_COUNT_HW_INSTRUCTIONS (an architected enum) or a raw PERF_TYPE_RAW config and asks no questions about what it means. The mapping from a human name (ex_ret_instr, INST_RETIRED.ANY) to those numbers is a per-microarchitecture table maintained outside the kernel — libpfm4, LIKWID, intel/perfmon — and is event-naming.md's whole subject. The kernel exposes config freely to unprivileged processes (subject to privilege gating), which — unlike Windows/macOS capability curation — is why Linux is the survey's "any event" outlier.

Strengths ​

  • One uniform acquisition hub. Counting and sampling are the same syscall and the same attr ABI; there is no second API to learn for profiling vs benchmarking. Windows and macOS both split this across several subsystems.
  • Atomic group scheduling makes derived metrics (IPC, MPKI) honest — the co-scheduled counts share one window or the group does not schedule at all.
  • User-space counter reads via rdpmc + the mmap page reach tens-of-cycles overhead, the natural fit for per-iteration bracketing inside a benchmark loop.
  • Self-describing sampling records. The sample_type bitmask makes each record's layout deterministic; druntime's core.sys.linux.perf_event already models the mmap page, headers, and every PERF_SAMPLE_*, so a pure-D sampler needs no C shim for the ring buffer (only libdwfl needs extern(C)).
  • Unprivileged by default (perf_event_paranoid permitting): any config value is openable without a global registration step.
  • A public, reference consumer (tools/perf) whose record → symbol pipeline is readable source, not a black box.

Weaknesses ​

  • The MMAP2/stale-binary hazard (concern 2): a naive self-profiler captures zero mmap records and must synthesize the address-space model from /proc/PID/maps and validate build-ids, or it symbolizes against the wrong binary silently.
  • Multiplexing error is real and unbounded below a millisecond of running time — a sub-ms slice can inflate the scale to 5×+ (Experiment A). The harness must detect rotation and shrink the group, not trust the estimate.
  • rdpmc capability is arch- and policy-dependent: cap_user_rdpmc is set by arch code and may be disabled by sysadmin policy; a portable path must probe it, never assume it.
  • The decode surface is four separate libraries, each with its own ABI and version skew — the acquisition is trivial, the interpretation is where the engineering lives.
  • Privilege gating is multi-dimensional: perf_event_paranoid levels, tracefs file permissions (concern 5), and SPE physical-address gating (concern 3) each fail differently and must be treated as runtime capability probes.

Key design decisions and trade-offs ​

DecisionRationaleTrade-off
One syscall for counting and sampling (perf_event_open)A single flat attr ABI covers the whole surface; nothing to unify at the consumerThe attr struct is wide and versioned; every capability is a bit or field to probe
All-or-nothing group schedulingCo-scheduled counts share one window → derived metrics are validA group larger than the counter budget silently does not schedule — sizing is the caller's job
time_enabled/time_running + a documented scaling formulaThe reader can recover a full-window estimate under multiplexingEstimates degrade sharply below a millisecond of running time; running == 0 is unrecoverable
rdpmc + a seqlock'd mmap page for user-space readsTens-of-cycles counter reads, no syscall, ideal for per-iteration bracketingCapability is arch-set and policy-gated; needs a seqcount retry loop and a probe
MMAP2 only for mappings made while enabledThe kernel avoids re-dumping the entire (large, static) address space per eventSelf-profilers capture nothing pre-existing → must synthesize from /proc/maps + validate build-ids
Decoders (libdwfl/libtraceevent/libnuma) live outside the kernelThe acquisition path stays minimal; decode complexity is opt-in and swappableFour ABIs, four version-skew surfaces, and none of them portable off Linux
config openable by unprivileged processes (paranoia-gated)No global event registration; any raw selector is reachableShifts the curation burden to userspace — a portable backend must advertise, not assume

Sources ​