Flame Graphs()

  本篇文章为你整理了Flame Graphs()的详细内容,包含有 Flame Graphs,希望能帮助你了解 Flame Graphs。

  Flame Graphs

  
CPU Flame Graph

  Flame graphs are a visualization of hierarchical data, created to visualize stack traces of profiled software so that the most frequent code-paths to be identified quickly and accurately. They can be generated using my open source programs on github.com/brendangregg/FlameGraph, which create interactive SVGs.

  My colleague on the Netflix performance engineering team, Martin Spier, created an open source d3 version: d3-flame-graph.

  See the Updates section for other implementations.

  The following pages (or posts) introduce different types of flame graphs:

  
The example on the right is a portion of a CPU flame graph, showing MySQL codepaths that are consuming CPU cycles, and by how much.

  Flame graphs can also be used for any hierarchical data. E.g., file system contents (see instructions; comparisons with treemaps and sunbursts).

  On this page: Summary,

  OSes,

  Presentation,

  Variations,

  Origin,

  Updates.

  Summary

  The x-axis shows the stack profile population, sorted alphabetically (it is not the passage of time), and the y-axis shows stack depth, counting from zero at the bottom. Each rectangle represents a stack frame. The wider a frame is is, the more often it was present in the stacks. The top edge shows what is on-CPU, and beneath it is its ancestry. Original flame graphs use random colors to help visually differentiate adjacent frames. Variations include inverting the y-axis (an "icicle graph"), changing the hue to indicate code type, and using a color spectrum to convey an additional dimension.

  Flame graphs are both a static and dynamic visualization. As a static visualization, a flame graph can be saved as an image, included in print (books), and will still convey the "big picture" as only the most frequent frames have enough width for labels. A dynamic visualization allows interactive features to aid navigation and comprehension, including:

  
Mouse hover shows additional frame details in a status bar.

  Mouse click zooms the visualization horizontally, revealing function names previously elided.

  Search matches and highlights a given term and shows the "cumulative percentage" of stacks including the search term.

  
Flame graphs can also be generated from any profile data that contains stack traces, including from the following profiling tools:

  
Once you have a profiler that can generate meaningful stacks, converting them into a flame graph is usually the easy step.

  There are also numerous profiling products and companies that now support flame graphs. See the Updates section below.

  Variations

  Icicle charts are flame graphs upside down. Some people prefer it that way. My flamegraph.pl creates them using --inverted. I prefer the standard "flame" layout, where the y-axis is counting stack depth upwards from zero at the bottom. Im also used to scanning them top-down to look for plateaus. But for very deep stacks the flame graph layout (with a GUI that starts at the top) often means the initial view may be mostly empty (a few thin interrupt stacks) forcing the developer to scroll down to find the bulk of the profile. For developers who prefer reading root-to-leaf anyway, an icicle layout instead means that the starting point is always on screen without needing to scroll. For that reason, many flame graph implementations use the icicle layout by default instead. Others use the flame graph layout but begin showing the bottom so that the root frames are on screen. I dont have a strong opinion about this, do whichever you prefer! Preferably include a toggle so that the end user can pick their preferred layout.

  Flame charts were first added by Google Chromes WebKit Web Inspector (bug). While inspired by flame graphs, flame charts put the passage of time on the x-axis instead of the alphabet. This means that time-based patterns can studied. Flame graphs reorder the x-axis samples alphabetically, which maximizes frame merging, and better shows the big picture of the profile. Multi-threaded applications cant be shown sensibly by a single flame chart, whereas they can with a flame graphs (a problem flame charts didnt need to deal with, since it was initially used for single-threaded JavaScript analysis). Both visualizations are useful, and tools should make both available if possible (e.g., TraceCompass does). Some analysis tools have implemented flame charts and mistakingly called them flame graphs.

  Sunburst layout using radial coordinates for the x-axis, a flame graph can be turned into a hierarchical pie chart. The Google Web Inspector team prototyped them. I also discussed them vs flame graphs in my comparison post.

  Origin

  I invented flame graphs when working on a MySQL performance issue and needed to understand CPU usage quickly and in depth. The regular profilers/tracers had produced walls of text, so I was exploring visualizations. I first traced CPU function calls and visualized it using Neelakanth Nadgirs time-ordered visualization for callstacks, which itself was inspired by Roch Bourbonnaiss CallStackAnalyzer and Jan Boerhouts vftrace. These look similar to flame graphs, but have the passage of time on the x-axis. But there were two problems: the overhead of function tracing was too high, perturbing the target, and the final visualization was too dense to read when spanning multiple seconds. I switched to timed sampling (profiling) to solve the overhead problem, but since the function flow is no longer known (sampling has gaps) I ditched time on the x-axis and reordered samples to maximize frame merging. It worked, the final visualization was much more readable. Neelakanth and Rochs visualizations used completely random colors to differentiate frames. I thought it looked nicer to narrow the color palette, and picked just warm colors initially as it explained why the CPUs were "hot" (busy). Since it resembled flames, it quickly became known as flame graphs.

  I described more detail of the original performance problem that led to flame graphs in my ACMQ/CACM article (link above). The flame graph visualization is really an adjacency diagram with an inverted icicle layout, which I used to visualize profiled stack traces.

  Updates

  Flame graphs were released in Dec 2011. Not long afterwards (updated in 2012):

  
Alan Coopersmith generated flame graphs of the X server.

  Dave Pacheco created them with node.js functions.

  Max Bruning has also shown how he used it to solve an IP scaling issue.

  Dave Pacheco has also created stackvis, a Node.js implementation, available on npm.

  Zoltan Farkas was inspired by flame graphs to create Spf4j (Simple performance framework for java), which includes them in a GUI.

  
I wrote some documentation for Linux Kernel Flame Graphs, generated using either perf and SystemTap for the profile data.

  Mark Probst developed Flame Graphs for Instruments on Mac OS X.

  Sam Saffron has developed Flame graphs in Ruby MiniProfiler, and shows examples of amazingly deep stacks.

  Bruce Dawson has an excellent post on Summarizing Xperf CPU Usage with Flame Graphs on Microsoft Windows. It includes examples for Visual Studio and Outlook, and a stack folding script to process the Xperf output.

  Google Chromes performance analysis tool, WebKit Web Inspector, introduced "Flame Charts", inspired by flame graphs. These are a similar visualization, but the x-axis is time, instead of the alphabet. Check out the screenshot, which includes a mouse-over popup that links to the source code. (Update: there is a bug to add flame graphs to Chrome, in addition to flame charts: Chromium 452624.)

  Tim Bunce has been improving and adding features to Flame Graphs, and has included them in Perls best profiler, Devel:::NYTProf, for profiling Perl. See his post on NYTProf v5 Flaming Precision.

  
I wrote a document summarizing four techniques for generating Memory Leak (and Growth) Flame Graphs, which visualize stacks with byte counts, instead of the traditional CPU sample Flame Graphs. I also colored them green to indicate that they are a different type.

  John Graham-Cumming showed how CloudFlare was using SystemTap-generated flame graphs for optimizing their Lua WAF.

  Yichun Zhang showed how Off-CPU Time Flame Graphs (PDF) can solve issues of blocking time. See Off-CPU Analysis for why this is important.

  Igor Soarez wrote How To Make Flame Graphs for node.js analysis, showing all steps involved.

  Paul Irish and Umar Hansa posted an awesome demo of using FlameCharts to investigate time in V8, which includes zooming in and clicking on functions to go to code, and Addy Osmani posted a longer video tutorial. While these arent Flame Graphs, they show developments in a related visualization: a time-series version that retains sequence and ordering.

  
I gave a talk at USENIX/LISA13 titled Blazing Performance with Flame Graphs, which covered them in enormous detail. See the slides and video. At the last minute this talk became a Plenary when another speaker became ill and had to cancel.

  Joab Jackson wrote an article about flame graphs for PCWorld, InfoWorld, and CIO.

  I wrote documents on Off-CPU Flame Graphs, and Hot/Cold Flame Graphs, based on what I had documented earlier for my USENIX talk.

  I wrote a post showing how the Google lightweight-java-profiler can be used to create Java Flame Graphs. This is only Java code time, and not other CPU consumers. See my later work with Java (-XX:+PreserveFramePointer) for a different and more complete profile.

  Vladimir Kirillov has developed eflame (github), a Flame Graph profiler for Erlang. Blocking calls are in blue, see the screenshot.

  Trevor Norris posted instructions for creating Linux Node.js CPU flame graphs, by using perf_events and --perf-basic-prof. See the example.

  Luca Canali provides many examples of Flame Graphs for Oracle databases, and explains what they mean.

  Julien Charbon posted some CPU flame graphs on FreeBSD, to analyze TCP stack lock contention with short-lived connections.

  Gabriel posted Profiling and optimising with Flamegraph, explaining all the steps with some really good examples.

  
Facebooks Strobelight uses an inverted flame graph for perf analysis. See the slide in Yannick Brosseaus Using tracing at Facebook scale (PDF) talk.

  The post on Profiling a Meteor app: Telescope includes flame graphs.

  Adrien Mahieux added the Click to Zoom feature to the original FlameGraph software. This is really awesome!

  I wrote a post on Differential Flame Graphs, showing new features I added to flame graphs for performance regression testing.

  I came up with an unusual but useful use for differential flame graphs: CPI Flame Graphs, to highlight memory stall cycles.

  I gave a talk at the 2014 FreeBSD Developer and Vendor Summit on Flame Graphs on FreeBSD.

  I wrote a post on Node.js Flame Graphs on Linux using Linux perf_events and v8s --perf-basic-prof option.

  Yunong Xiao showed how flame graphs helped solve an important production performance issue for Netflix in the post Node.js in Flames.

  Oozou developed and published RubyProf::FlameGraphPrinter for ruby-prof, to generate folded stacks for flame graph generation.

  Thorsten Lorenz developed a web flamegraph interface that can load new profiles, and perform a regexp search. See his example.

  
Shawn Sterling posted about his excellent talk at Linuxfest Northwest 2014: Getting Started With Flamegraph, which includes both the slides and video (turn the volume up).

  Scott Lystig Fritchie gave a great talk on profiling Erlang code including flame graphs at Erlang Factory San Francisco 2015.

  I submitted tickets for a flame graph / flame chart toggle for Google Chrome (Issue 452624) and Firefox (bug 1123495). Id love to see both implemented. The chromium ticket prompted interesting discussion and prototypes.

  In my SCALE13x talk (2015), I previewed mixed-mode Java flame graphs using Linux perf_events ("perf"). This uses a JVM frame pointer patch that has become the -XX:+PreserveFramePointer option in both JDK9 (JDK-8068945) and JDK8u60 (JDK-8072465). For the first time, we can see all CPU consumers in one visualization! See slides 40 to 57 for more details, and my post about it (when I write it!).

  Francesco Mazzoli post about Flame graphs for GHC time profiles with ghc-prof-flamegraph, including a tutorial for making flame graphs from the existing GHC (Haskell) profiler output.

  Robin Moffatt posted about Analysing ODI performance with Flame Graphs, for understanding load plans in the Oracle Data Integrator.

  Flame graphs were demoed at d3NYC (Drupal) 2015, but I havent found slides online yet.

  Cor-Paul Bezemer has been investigating flame graph differentials with his Flamegraphdiff software, which shows the difference from A to B using three flame graphs simultaneously. This was also the subject of a SANER2015 paper ($), and talk.

  M. Isuru Tharanga Chrishantha Perera wrote a translator for Java Flight Recorder profiles, for making flame graphs.

  Jan St pie developed Pluggable Flame Graphs for Clojure (example).

  Evan Hempel has created a python-flamegraph profiler for Python, which generates the folded stack output suitable for making into flame graphs. See the README on github for instructions.

  Franck Pachot posted another example of using CPU flame graphs for identification of an Oracle database bug.

  Strongloop have included a nice looking Node.js flame graphs module for their Arc Profiler.

  @yoheia posted a detailed flame graph post showing Linux kernel profiling (in Japanese).

  Zoltan Majo fixed JDK-8068945 for Java 9 and JDK-8072465 for Java 8 update 60 build 19 (or later, download as early access here). This adds -XX:+PreserveFramePointer, which allows Linux perf_events to sample full stacks for making flame graphs. This began with a prototype patch I developed and submitted: (A hotspot patch for stack profiling). Great to see this functionality make it into the JVM!

  Min Zhou has developed PerfJ, for automating the collection of Java flame graphs, using the frame pointer patch (see previous item).

  
Myself and Martin Spier posted about Java in Flames (PDF) for the Netflix Tech Blog, showing Java mixed-mode flame graphs using the new -XX:+PreserveFramePointer JVM option.

  Jonathan Perkin used memory flame graphs for Reducing RAM usage in pkgin.

  Isuru Perera postabout about Java CPU Flame Graphs with an example making use of Mins PerfJ.

  Ben Sandler from Uber has posted go-torch, a flame graph profiler for go-lang programs.

  Andi Kleen demonstrated Generating flame graphs with Processor Trace, a feature from modern Intel CPUs for very high frequency sampling.

  I wrote about the new flame graph search feature. The matched percentage is very handy, so I dont have to mouse over many tiny frames and add them up manually.

  Eben Freeman posted about Profiling Python in Production, which includes a python profiler and basic d3 flame graphs.

  Will Sewell mentioned flame graphs in his Top tips and tools for optimising Haskell post, and linked to the work for GHC flame graphs by Francesco Mazzoli.

  Bo Lopker created djdt-flamegraph (github) for getting a flame graph of current requests in the Django Python web framework.

  NodeSource have flamegraphs in their NSolid Node.js analysis product. The graphics look very nice, and they also have treemaps and sunbursts (both of which I think are usually less effective than flame graphs, but I dont mind having the option).

  Alex Ciminian has been developing a d3-flame-graphs library (github) for d3, implemented in CoffeeScript. Check out the demo, it looks really nice so far.

  Martin Spier (a colleague at Netflix) has been developing d3-flame-graph, as a d3 library implemented in JavaScript. This will be integrated into our open source Vector instance analysis tool. It has zoom transitions!

  I gave a Java Mixed-Mode Flame Graphs talk at JavaOne 2015, including all the latest updates.

  David Calavera wrote about Docker flame graphs for Go.

  Carol Nichols wrote detailed instructions for Rust Profiling with Instruments and FlameGraph on OSX: CPU/Time.

  
Apekshit Sharma wrote a post about Saving CPU! Using Native Hadoop Libraries for CRC computation in HBase.

  Mike Huang wrote a Netflx tech blog post to show how he helped with Saving 13 Million Computational Minutes per Day with Flame Graphs.

  Srdjan Marinovic and Ryan Day created goprofui includes a cpu.go profiler for golang and flame graphs.

  I wrote an article for ACMQ, The Flame Graph, which defines flame graphs, describes their origin, explains how to interpret them, and discusses possible future developments.

  David Mark Clements has developed 0x, a new interactive flame graph profiler for Node.js processes on both Linux and OS X. See the demo.

  Qt Creator 4.0.0 now includes flame graphs along with timeline and statistical views. Thats how it should be done: different views of the same data you can switch between.

  Antonio P rez developed 4gl-flamegraph for processing the profiler output of Genero by 4Js (Four Js).

  Ty Overby published a flamegraph profiling tool for rust.

  The vprof Python package provides interactive visualizations for profiling, including flame graphs.

  Alastair Butler and Kei Yoshimoto published the paper Large scale semantic representation with flame graphs (PDF), where they used them to visualize semantically annotated corpora, which has been parsed into a hierarchy.

  Jonathan Newbrough developed the Gumshoe Load Investigator for Java analysis, initially for internal use in the Dell Cloud Manager, which makes good use of flame graphs.

  Mahesh Dathrika published Igniting Node.js Flame, showing how they use the v8-profiler at eBPF to create Node.js flame graphs.

  Maciek Lesiczka is building an ASP.NET profiler that includes flame graphs, and introduced it in netric.io - lightweight ASP.NET profiler.

  Nitsan Wakart covered Java flame graphs and various Java profiling challenges in his Devoxx UK talk Extreme Profiling: Digging Into Hotspots by Nitsan Wakart (youtube).

  Maciek Lesiczka has been developing netric.io - lightweight ASP.NET profiler, an ASP.NET profiler that includes flame graphs.

  My article The Flame Graph was published in CACM.

  
Rhys Hiltner from Twitch blogged about Gos march to low-latency GC, which included flame graph analysis of GC time.

  Dmytro Semenov on the ebay tech blog posted Mastering the Fire, about how they can create flame graphs on production servers at any time with one click of a button.

  Olivier Cano posted about Measuring Snap performance which includes flame graphs for golang.

  Kay Ousterhout wrote about Generating Flame Graphs for Apache Spark using Java Flight Recorder.

  Adam Perry published Rust Performance: A story featuring perf and flamegraph on Linux, which also has great examples of using perf.

  Bruce Dawson published a post on ETW Flame Graphs Made Easy and the new support in Windows Performance Analyzer (WPA).

  Aviem Zur from PayPal engineering wrote about Spark in Flames - Profiling Spark Applications Using Flame Graphs.

  C dric Champeau (I think) has added automatic Flame graph generation for Gradles perf tests. Makes a lot of sense, as does auto generation for software versions and builds (given a load generator).

  Bill Smith from indeed engineering posted A Funny Thing Happened on the Way to Java 8, where code cache issues are investigated with flame graphs (Ive also found a code cache issue this way!).

  Bert Hubert included flame graphs in his detailed analysis Optimizing optimizing: some insights that led to a 400% speedup of PowerDNS.

  Luca Canali has posted more flame graphs, this time Apache Spark 2.0 Performance Improvements Investigated With Flame Graphs, and even includes some perf stat analysis.

  Mahmoud Hatem posted about perf_events : Off/On/Mixed CPU flamegraph extended with oracle wait events, showing both on- and off-CPU flame graphs for an Oracle workload.

  Evan Klitzke posted pyflame: Uber enginerings ptracing profiler for python, as well as the source on github. Python is tricky to profile as its interpreted only, so basic frame pointer-based stack walking only identifies interpreter frames.

  I posted about Linux 4.9s Efficient BPF-based Profiler, an important feature that will make flame graph profiling much more efficient.

  Jerome Terry posted instructions for Generating Java Mixed Mode Flame Graphs summarizing all the latest steps.

  Joel Fernandes posted ARMv8: Flamegraph and NMI Support.

  
Linkedin have done some great work with flame graphs: ODP: An Infrastructure for On-Demand Service Profiling, including searching and comparison (differential) capabilities.

  Alice Goldfuss posted about Making FlameGraphs with Containerized Java, figuring out how to get perf to work in a container environment.

  Nitsan Wakart wrote a great post on Java Flame Graphs Introduction: Fire For Everyone!, that explains different Java profilers in depth.

  Benoit Bernard posted Using Ubers Pyflame and Logs to Tackle Scaling Issues showing all steps.

  I posted Where has my disk space gone? Flame graphs for file systems, with a follow up Flame Graphs vs Tree Maps vs Sunburst.

  Sasha Goldshtein posted Profiling a .NET Core Application on Linux where he solves problems and discusses next challenges with profiling Windows .NET on Linux.

  Steve Robinson posted Making sense out of flamegraphs (Ruby on Rails) with an introduction to stack sampling profilers.

  Ross Schlaikjer posted Profiling Android apps with Flamegraphs, and has an online Android Trace file to flamegraph converter.

  Claes Redestad posted Bytestacks visualizing the output of -XX:+TraceBytecodes

  Lari Hotari has created jfr-report-tool to turn a Java Flight Recorder (JFR) dump into either CPU flame graphs or allocation flame graphs.

  Nan Xiao posted Use perf and FlameGraph to profile program on Linux, including a sample C++ program to analyze.

  Michael Malis at Heap posted How Basic Performance Analysis Saved Us Millions, with flame graph analysis of Postgres.

  I posted Java Package Flame Graph, showing this new way to inspect Java CPU usage.

  
Jon Hadad wrote about Analyzing Cassandra Performance with Flame Graphs.

  Amir Langer gave a talk (in Hebrew) about Flame Graphs and the JVM at eBay (youtube).

  Datadog say they added flame graphs for Java performance but they look like flame charts. Im not sure yet.

  Brendan Ryan wrote about Profiling Go Applications with Flamegraphs using Ubers go-torch library.

  Kiran posted How to use linux perf tools and save dollars - Part 2, including flame graphs of perf samples.

  Vladimir Agafonkin developed Flamebearer, for lightweight responsive graphs for Node.js and v8.

  Trace Compass added flame graphs as well as flame charts, the first tool to support both (yay).

  Myself and my Netflix colleague Martin Spier published FlameScope, a new performance analysis tool that allows profiles to be visualized as subsecond-offset heat maps, and then time ranges to be selected and then visualized as a flame graph. Its also on github Netflix/flamescope.

  Sadiq Jaffer, Richard Warburton wrote about Always-on production Flame Graphs for Java.

  Alexandru Olaru posted Squeeze node.js performance with flame graphs.

  Ivan Babrou posted a case study of https://blog.cloudflare.com/tracing-system-cpu-on-debian-stretch/, including flame graphs (and bcc/eBPF tools).

  Marcus Hirt developed jmc-flame-view and posted instructions for getting it going: this adds flame graphs to Java Mission Control.

  Jamie Wong has developed SpeedScope - Interactive Flamegraph Explorer, which supports "time order" (flame charts), "left heavy", which are flame graphs with an additional sort so the largest frames move left, and "sandwich" ("middle-out" merge). Great to see all three types in one GUI.

  Sebasti n Reca (Muun) showed at at Kaizen how flame graphs helped with Scaling Bitcoin (youtube video).

  
Amazon launched the AWS CodeGuru profiler that includes flame graphs. Also see the flame graph examples in Optimizing application performance with Amazon CodeGuru Profiler.

  Google added flame graphs to Google Cloud Profiler. By default these use a largest-frame-left sort (Id rather alphabetic so that towers dont switch positions when comparing two profiles from the same system). There are flame graph examples in How Mercari reduced request latency by 15% with Cloud Profiler.

  
Thanks to everyone who has written about flame graphs, developed them further, and shared their results! Ill update this page from time to time with more news.

  以上就是Flame Graphs()的详细内容,想要了解更多 Flame Graphs的内容,请持续关注盛行IT软件开发工作室。

郑重声明:本文由网友发布,不代表盛行IT的观点,版权归原作者所有,仅为传播更多信息之目的,如有侵权请联系,我们将第一时间修改或删除,多谢。

留言与评论(共有 条评论)
   
验证码: