bpftrace is a microscope, not a dashboard
A handful of one-liners that answer "who is doing this syscall, on what file, right now" without touching the kernel module machinery.
- problem
- Production troubleshooting often needs a live answer about a specific process, file, or syscall path, but attaching a full eBPF program or a ftrace session is heavyweight when the question is narrow.
- scope
- A working set of bpftrace one-liners for syscall attribution, file-access tracing, latency histograms, and kernel function argument capture, with the caveat that each probe has a cost.
- environment
- Linux hosts with bpftrace installed, BTF available, and CAP_BPF or root access; used for ad-hoc diagnosis on bare metal, VMs, and containers.
Assumptions
- The kernel is recent enough to expose BTF and allow tracepoint and kprobe attachment.
- The operator is diagnosing a reproducible or live condition, not capturing a one-in-a-million race for offline analysis.
- Probe cost is acceptable for the duration of the trace; bpftrace is not a permanent collector.
Limitations
- One-liners are diagnostic tools, not a replacement for a structured OpenTelemetry pipeline in steady state.
- kprobe names and argument offsets vary across kernel versions; verify with the running kernel headers or BTF dump.
- Some distributions restrict unprivileged bpf; the privilege boundary is itself a security decision.
Table of contents 4 sections
Attribute a syscall to a process
The first question is usually "who is calling this." A tracepoint on the syscall enter path prints the process name, pid, and the syscall argument every time the target fires. This is the one-liner equivalent of attaching strace to every process simultaneously, filtered to one syscall.
The cost is proportional to how often the syscall fires. A rare syscall like accept on an idle service is nearly free. A hot path like read on a database will produce thousands of events per second; add a count aggregation instead of printing every hit.
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%-16s %-7d %s\n", comm, pid, str(args->filename));}'Build a latency histogram without a metric pipeline
A latency question does not need a metric server. A kprobe on the function entry paired with a kretprobe on the return measures the elapsed nanoseconds, and a power-of-two histogram bucket summarizes the distribution in one line. The result is a text histogram printed on Ctrl-C.
This works for kernel functions, VFS operations, block IO, and any function with a stable entry and return. The distribution shape is often more useful than a single p50 number: a bimodal histogram reveals two populations, which a dashboard average would flatten into one misleading bar.
bpftrace -e 'kprobe:vfs_read { @[pid, comm] = nsecs; }kretprobe:vfs_read /@[pid, comm]/ { @lat = hist(nsecs - @[pid, comm]); delete(@[pid, comm]);}'Aggregate instead of printing
The difference between a one-liner that runs for five minutes and one that cripples the host is the output strategy. A printf on a hot syscall writes to user space per event. An @-aggregation collects in kernel memory and prints a summary on exit. For anything that fires more than a few hundred times per second, switch to a count or hist aggregation before the trace runs long.
A useful pattern is to count by process name: @counts[comm] = count(). This answers "which process is responsible for the load" in a single table without per-event output. The aggregation lives in BPF maps; clearing it with clear(@counts) resets the window without restarting the trace.
Accept that probes are kernel-version-shaped
A kprobe by function name can vanish across kernel versions. A tracepoint is more stable but covers a smaller surface. When a one-liner fails to attach, the first check is whether the kernel still exposes that symbol, not whether the syntax is wrong.
BTF makes this tractable: bpftrace -lv tracepoint:syscalls:* lists available tracepoints on the running kernel, and bpftrace -lv kprobe:vfs_read shows argument types. Verifying against the live kernel before running is faster than debugging a "probe not found" error on a production host under pressure.
- field-observed Reproducible workshop usage
These one-liners have answered real "who opened this file" and "what is the latency distribution of this syscall" questions on production hosts without writing a kernel module.
- reference bpftrace reference guideopen ↗
Official language reference, probe types, builtins, and one-liner collection.