Server profiles and debug symbols

What a whole machine is doing, and naming a stripped binary's functions.

Server profiles and debug symbols

Profiling a whole server

A Service's profile answers "what is my application doing". A server profile answers "what is this machine doing": every process on it, whoever started it.

Open a server and choose its Performance tab. It shows whether the server can be profiled (the same requirements as a Service; see Performance profiling) and its busiest processes right now, such as gunicorn 96% CPU. Those figures are averages since each process started, as ps reports them; a profile shows what they do now.

A server profile can answer five of the questions:

What is using the CPU
Where time goes, including waiting
What every thread is doing right now
What is waiting on locks
Where memory grows

What is holding memory and Where memory is allocated are not offered, because they are read from inside one runtime at a time. Capture them from the Service instead.

Nothing is signalled

A server profile samples every process with eBPF in the kernel and does nothing else. In particular, it never opens the Node.js inspector, which a Service profile does for Node.js 24 and newer. For those processes it keeps what eBPF can read, which names the native frames but not the JavaScript ones. Profile the Service itself to see its JavaScript.

Grouped by what owns each process

The first row of a server profile is what each process belongs to, and the processes sit beneath:

Group	What it is
Service billing	One of your Services, by name
Container redis	A container that is not one of your Services
Unit nginx.service	A systemd unit that is not one of your Services
Signed-in sessions	Anything started from a login session or a shell
System	The kernel's own threads and the init system
Other processes	Anything SlideOps could not place

The answer says which group used the most and what inside it, for example:

Over 30 seconds, Service billing used 54% of the CPU time on this server, most of it in build_report. Next was Unit nginx.service, at 14%.

Colour by: Language colours every box by the language it is in, which on a mixed server shows at a glance what is Python, Java, Go or native code.

Server profiles are listed under Past profiles on the server's Performance tab and on the Profiles page, and two of the same type can be compared.

Removing the profiler

The profiler stays installed so the next capture starts quickly. Remove the profiler from this server, on the server's Performance tab, deletes it and any capture files left behind. The next capture installs it again.

Naming a stripped binary's functions

A release build is often stripped: it runs the same, but carries no function names, so the profiler can only show an address, such as app+0x4a1f20. Go built with -ldflags=-s, and Rust or C release builds without symbols, are the usual cases.

When stripped binaries you ship hold 1% or more of a profile, a card at the top of the profile names them:

Some functions have no names, because their binaries were stripped /usr/local/bin/billing-worker 31% of this profile, build b65a313cbf47

The language runtimes themselves and system libraries are never listed, even when stripped: their internals already sit under the function of yours that called them.

Uploading debug symbols

Press Upload debug symbols on that card, or in the Debug symbols panel at the bottom of any Performance tab, and choose one of:

the unstripped build of the same binary, or
the debug file made from it before stripping.

Keeping a debug file aside is one step in a release build:

objcopy --only-keep-debug app app.debug
strip app

Ship app, and upload app.debug.

SlideOps reads only the function names and addresses from the file. The file itself is never stored. The names are kept for the Workspace, and from then on they name that build's functions in every profile of it, including profiles taken before the upload: reopen one and its addresses have become names.

Files up to 1 GB are accepted. Uploading the same build again replaces the earlier upload. The Debug symbols panel lists every upload with its file name, how many functions it named and its build, and lets you delete one.

Matching by build id

An upload is matched to a binary by its build id, a fingerprint the linker writes into each build. That is what makes it safe: a debug file can only ever name the exact build it came from, never a different version that happens to share a file name.

gcc, clang and Rust write a build id by default. Go does not when it links internally; add -ldflags=-B=gobuildid to the build to have one.

If an upload is refused, the message says why:

Message	What to do
this is not an ELF binary or debug file	Upload the Linux binary or its `.debug` file, not an archive or a source file
the file has no build id, so SlideOps cannot tell which build it names	Rebuild with a build id, as above
the file is stripped too	Upload the unstripped build, or the debug file made from it

Profiles captured before SlideOps kept build ids say so on the card: capture again, then upload.