Linux Performance Tools, Brendan Gregg, part 1 of 2

Tutorial by Brendan Gregg of Netflix for O'Reilly Velocity conference 2015 Santa Clara. Part 1 of 2.
Slides: www.slideshare.net/brendangreg... or velocityconf.com/devops-web-pe...
Description: "
There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This tutorial explains methodologies for using these tools, and provides a tour of four tool types: observability, benchmarking, tuning, and static tuning. Many tools will be discussed, including top, iostat, tcpdump, sar, perf_events, ftrace, SystemTap, sysdig, and others, as well observability frameworks in the Linux kernel: PMCs, tracepoints, kprobes, and uprobes.
This tutorial is updated and extended on an earlier talk that summarizes the Linux performance tool landscape. The value of this tutorial is not just learning that these tools exist and what they do, but hearing when and how they are used by a performance engineer to solve real world problems - important context that is typically not included in the standard documentation."

[@domaincontroller]

07:08 system is running slow, commands
07:11 top (10)
07:31 iotop
07:45 iostat -xz 1 (6)
07:57 netstat
08:14 ss
08:34 dstat
08:43 sar -n DEV 1 (8)
08:59 vmstat (3)
09:10 | 09:49 strace
10:17 what do they mean by the system is slow, latency
11:51 packages, sysstat, procps, coreutils
============================================
12:07 Anti-methodologies
============================================
12:13 street light anti-method
13:25 drunk man anti-method
13:59 blame someone else anti-method
14:43 | 16:02 problem statement method
============================================
16:30 workload characterization method
17:48 USE method, utilization, saturation, errors
18:58 USE method for hardware systems
21:20 off-CPU Analysis
22:36 CPU profile method
24:26 RTFM Method
24:54 reading linux source code, jvm
26:50 How do you measure them
######################
27:07 observability tools
######################
27:18 uptime (1)
30:02 top (or htop)
31:57 ps -ef
32:28 vmstat 1 (3)
33:28 instant-xmdz
36:02 mpstat -P ALL 1 (4)
36:14 free -m (7)
36:46 latency is much higher, can you debug it
42:32 observability tools: basic
***********************
intermediate
***********************
42:48 strace
44:21 tcpdump
46:15 net stat, nicstat
46:49 pidstat 1 (5)
47:21 swapon
47:33 lsof
47:58 sar (8)
50:38 app is taking forever
50:45 | 51:17 pidstat, system time
50:57 system time, iostat
51:26 am i swapping ? vmstat
51:51 strace

[@JoJo-fy2vb]

7:46 iostat -x 1

[@yangroger1216]

thx

[@ranjansantra]

Thanks for this list!

[@Locane256]

Oh hey! You're the author of that observability tools image I have copied to every possible device and storage location so I can always reference it! Hell yeah, thank you!

[@TirajAdikari]

I'm troubleshooting a java performance issue at the moment and found this systematic method of analysis very useful. Thanks Brendan.

[@bhanureddy8780]

This is amazing. true love for the bits and bytes

[@abdullahadel1141]

Thank you for a wonderful session, much appreciated. 👏

[@ruoxizhang9486]

So clear and educational, thanks for sharing!

[@umamaheshsukamanchi]

Outstanding content. Very helpful

[@jaashutosh]

Very Useful content, thank you Brendan

[@OryBand]

One of the best devops presentations out there. Good job Brendan! You're making the world a better place one step at a time.

[@rhugga]

Performance tuning is not dev ops. This is basic, fundamental systems administration.

[@atom_rr]

@@rhugga hope that he knows what devops is nowadays...

[@subhabera5775]

Thanks Brendan :) You are awesome.

[@supremepancakes4388]

I like how you start off straight to the point when you said that when you are mentored by someone you basically eliminate what you dont need to know lol

[@karthiknedunchezhiyan1171]

Very informative! keep going!

[@charle09111]

OMG!! I really need it.

[@bullishtradersoul]

Excellent

[@ArslanFarooqQureshi]

You sir, are a legend.

[@moustaphakourouma9402]

Very useful thank you !

[@AndreKhan716]

Thank you!

[@shankerlee]

brilliant

[@dineshrathee3306]

Perfection !!

[@davidleal714]

Thank you man, I will definitely copy yours methods

[@king0s]

This is golden

[@bimalacharya9020]

fantastic !

[@azufendusgarendum6583]

Thank you very much

[@anglojojo]

Good video :)

[@pixelPlex]

Which tools are used for monitoring network usage/performance of a process?

[@harshbhadoria7200]

Hi guys, can anyone tell how is it possible to collect all metrics from Linux and visualize them in windows OS? How to make a bridge or connection between Linux and Windows and what are the technologies to be used.

[@yunuskoning7584]

Hi Brendan,
You are not by any chance available for freelance performance trouble shooting projects?

[@elabeddhahbi3301]

I'm speechless than you

[@QuadDerrick]

"more than 80 characters is a sin in unix world" :):):) didnt know but yea, makes sence.
Norway

[@sealovers1374]

they present linux but all of them use MacOS

[@retropaganda8442]

wtf is this macintosh :(

[@chenyifa]

too slow

[@sigridopps3049]

unprepared and confused

[@ManiBalajiC]

thats why he is the head of the netflix system performance.

[@rchaudry]

Thank you!