Reading a flame graph
The answer, hot spots, colours, zoom, search and every question a profile asks.
Reading a flame graph
A profile page answers before it asks anything of you. The top says in one sentence where the time went; the graph below is for going deeper. This page walks through it from top to bottom.
The answer
The first card is one sentence written from the profile itself, for example:
Over 30 seconds, 62% of the CPU time was in build_report, the largest part inside dumps (24%).
It names the function in your own code that the time went to, because that is the code you can change, and the hottest thing inside it when that is a real share of the whole. The sentence adapts to the question asked:
| Profile | The sentence says |
|---|---|
| What is using the CPU | how much of the CPU time was in a function |
| Where time goes, including waiting | how much of the elapsed time, waiting included |
| What every thread is doing right now | how many threads are mostly running and how many mostly waiting, and where |
| What is waiting on locks | how much of the waiting was under a function |
| What is holding memory | how much of the memory in use was allocated in a function, when the capture ended |
| Where memory is allocated | how much of the memory allocated came from a function |
| Where memory grows | how many of the requests for more memory came from a function |
An empty profile says what that means: "Nothing waited on a lock in these 30 seconds", or "The Service did not ask for more memory in these 30 seconds: its memory was steady."
Below it, any notes from verification, then What to look at (see Profiles and recommendations), and the AI summary when your deployment has one.
Hot spots
Three cards name the functions that matter most. Your own functions come first, each with the share it is answerable for: its own time plus the library and runtime code it called, but not your other functions it called. When your code accounts for little, the functions with the most time of their own fill the rest.
Each card shows the share, the function, its language, file and line, and whether it is your code. Click one to find it in the graph.
The graph
- Width is time (or bytes, or requests, for the memory questions). Look for the wide boxes; height does not matter.
- Read it from the bottom up. The bottom row is where the program started, and each box is called by the box directly below it, so a tower is a chain of calls.
- The first row above all is each process, so a Service with several processes keeps them apart.
A first visit opens a three-step guide that says the same. How to read this brings it back at any time.
Colour says whose code it is
| Colour | Whose code |
|---|---|
| Orange | Your code: functions in your application |
| Blue | Libraries: packages your application imports |
| Violet | Runtime: the language runtime and its standard library |
| Grey | System: the Linux kernel and the C library |
| Pale grey | Waiting: a thread waiting for the network, a lock, a timer or disk |
The legend above the graph is also a filter: click Libraries to fade them, and click again to bring them back. The right of the legend gives each language's share of the profile.
A whole-server profile adds Colour by: Whose code / Language, which colours every box by the language it is in instead.
Hover, click and the keyboard
- Hover a box for its function, language, file and line, its share including what it called, and its share on its own.
- Click a box to zoom in so it fills the width. The path back appears above the graph; click any step of it, or the box again, to zoom out. Reset zoom returns to the whole graph.
- Keyboard: the arrow keys move between boxes (up into the widest box it calls, down to its caller, left and right to its neighbours), Enter zooms, Escape zooms out.
Search
Type in Search functions, or press / from anywhere on the page. Every matching box lights up, everything else fades, and the toolbar says how many boxes match and what share of all the time they hold together.
Flame or icicle
Flame grows upward from where the program started. Icicle hangs downward, which is easier for very deep stacks.
What is folded away
Two things are folded so the graph reads cleanly:
- Small functions. Boxes under 0.1% of the whole are merged into one box per caller, such as "18 small functions". Hover it to see how many it stands for.
- Startup chains. The run of runtime and system frames every program starts with, and runs of unnamed native frames, collapse into one labelled box such as "13 runtime frames: _start … <interpreter trampoline>". Only a chain where each frame passes all of its time to one child is folded, so nothing is hidden but repetition.
Tick Show every frame to unfold the chains.
Unnamed frames
A box named like libc.so.6+0x29ca7 is native code no symbol covered: the binary was stripped. When stripped binaries you ship hold a real share of a profile, a card at the top names them. See Server profiles and debug symbols to name them.
Who calls it, and what it calls
Select any box and a panel opens under the graph with two small graphs for that function, merged from everywhere it appears:
- Called by: every path that led to it.
- Calls: everything it called.
This is the quickest way to see whether a function is slow everywhere, or only when one particular caller uses it.
Top functions
The table at the bottom lists the same data as the graph, so it is a complete alternative to it.
- Self is what the function itself accounts for; total includes everything it called.
- Your code shows only your functions; Everything shows all of them.
- Click a row to find the function in the graph.
Reading each question
Where time goes, including waiting. Waiting appears as pale grey boxes named for what the thread waited in, such as futex_wait or tcp_recvmsg. Wide waiting under one of your functions means that function spends its time waiting, not computing: on a database, the network, a lock or a timer.
What every thread is doing right now. One box per thread, all the same width, under its process. Inside, you see where each thread spent the 5 seconds. A thread that was blocked the whole time ends in (blocked the whole snapshot), with the kernel function it is blocked in beneath, which is how a hung thread shows.
What is waiting on locks. Only waiting on locks and conditions is counted: a futex, a mutex, a condition variable, Python's GIL or a Java monitor. The widest boxes are the code that waits most.
What is holding memory and Where memory is allocated. Width is bytes. Look for your function nearest the top of a wide tower: that is where the memory was allocated.
Where memory grows. Width is how many times the process asked the kernel for more memory, and the graph shows which code asked. A steady Service asks rarely; a growing one asks from the same place again and again.