并发与异步编程（三） -- 性能分析工具：gperftools和火焰图

1
2
3
4
5
6
1) Link your executable with -lprofiler
2) Run your executable with the CPUPROFILE environment var set:
     $ CPUPROFILE=/tmp/prof.out <path/to/binary> [binary args]
3) Run pprof to analyze the CPU usage
     $ pprof <path/to/binary> /tmp/prof.out      # -pg-like text output
     $ pprof --gv <path/to/binary> /tmp/prof.out # really cool graphical output

1
2
3
4
5
6
7
1) Link your executable with -ltcmalloc
    # 如果编译时没有-l链接，也可以在执行时使用 LD_PRELOAD
2) Run your executable with the HEAPPROFILE environment var set:
     $ HEAPPROFILE=/tmp/heapprof <path/to/binary> [binary args]
3) Run pprof to analyze the heap usage
     $ pprof <path/to/binary> /tmp/heapprof.0045.heap  # run 'ls' to see options
     $ pprof --gv <path/to/binary> /tmp/heapprof.0045.heap

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[CentOS-root@xdlinux ➜ FlameGraph git:(master) ]$ ./flamegraph.pl -h
Option h is ambiguous (hash, height, help)
USAGE: ./flamegraph.pl [options] infile > outfile.svg

    --title TEXT     # change title text
    --subtitle TEXT  # second level title (optional)
    --width NUM      # width of image (default 1200)
    --height NUM     # height of each frame (default 16)
    --minwidth NUM   # omit smaller functions. In pixels or use "%" for
                     # percentage of time (default 0.1 pixels)
    --fonttype FONT  # font type (default "Verdana")
    --fontsize NUM   # font size (default 12)
    --countname TEXT # count type label (default "samples")
    --nametype TEXT  # name type label (default "Function:")
    --colors PALETTE # set color palette. choices are: hot (default), mem,
                     # io, wakeup, chain, java, js, perl, red, green, blue,
                     # aqua, yellow, purple, orange
    --bgcolors COLOR # set background colors. gradient choices are yellow
                     # (default), blue, green, grey; flat colors use "#rrggbb"
    --hash           # colors are keyed by function name hash
    --random         # colors are randomly generated
    --cp             # use consistent palette (palette.map)
    --reverse        # generate stack-reversed flame graph
    --inverted       # icicle graph
    --flamechart     # produce a flame chart (sort by time, do not merge stacks)
    --negate         # switch differential hues (blue<->red)
    --notes TEXT     # add notes comment in SVG (for debugging)
    --help           # this message

1
2
3
4
5
6
7
8
[CentOS-root@xdlinux ➜ flamegraph_sample git:(main) ✗ ]$ stress --cpu 2 --io 2 --vm 2 --hdd 2 --timeout 5s
stress: info: [33388] dispatching hogs: 2 cpu, 2 io, 2 vm, 2 hdd
stress: info: [33388] successful run completed in 6s

# 另一个终端进行perf采集：
perf record -a -g
# 生成火焰图
perf script -i perf.data| stackcollapse-perf.pl | flamegraph.pl --reverse --inverted > stress_2cpu_2io_2vm_2hdd_icicle.svg

1
2
3
4
5
6
7
8
9
# stress模拟压力
stress --cpu 2 --io 2 --vm 2 --hdd 2 --timeout 5s

# bcc tools 采集堆栈
/usr/share/bcc/tools/offcputime -df > out.stacks
# 生成火焰图
flamegraph.pl --color=io --title="Off-CPU Time Flame Graph" --countname=us < out.stacks > offcpu_stress_2cpu_2io_2vm_2hdd.svg
# 冰柱型、反转堆栈
flamegraph.pl --color=io --title="Off-CPU Time Flame Graph" --countname=us --reverse --inverted < out.stacks > offcpu_stress_2cpu_2io_2vm_2hdd_icicle.svg

1
2
3
4
5
6
# bcc libbpf-tools
# /home/workspace/bcc/libbpf-tools/offcputime
offcputime > out.stacks
stackcollapse.pl < out.stacks | flamegraph.pl --color=io --title="Off-CPU Time Flame Graph" --countname=us > offcpu_stress2.svg
# 冰柱型、反转堆栈
stackcollapse.pl < out.stacks | flamegraph.pl --color=io --title="Off-CPU Time Flame Graph" --countname=us --reverse --inverted > offcpu_stress2_icicle.svg

1
2
3
4
/usr/share/bcc/tools/wakeuptime -f > out.stacks
flamegraph.pl --color=wakeup --title="Wakeup Time Flame Graph" --countname=us < out.stacks > wakeup_stress_out.svg
# 冰柱型、反转堆栈
flamegraph.pl --color=wakeup --title="Wakeup Time Flame Graph" --countname=us --reverse --inverted < out.stacks > wakeup_stress_out_icicle.svg

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# bcc tools采集，/usr/share/bcc/tools/wakeuptime
[CentOS-root@xdlinux ➜ bin git:(main) ]$ /usr/share/bcc/tools/wakeuptime 
Tracing blocked time (us) by kernel stack... Hit Ctrl-C to end.
^C
    # 被唤醒者
    target:          mysqld
    # 从上往下，是waker（即swapper/1）经过的堆栈
    # 辅助 CPU 启动的函数
    ffffffff97a00107 secondary_startup_64_no_verify
    # 进一步的 CPU 启动函数，负责启动辅助 CPU 的后续步骤
    ffffffff97a5929b start_secondary
    # CPU 启动的入口函数，进行一些初始化工作
    ffffffff97b22c7f cpu_startup_entry
    ffffffff97b22a84 do_idle
    ffffffff9814393c cpuidle_enter
    ffffffff981435fb cpuidle_enter_state
    # 处理APIC定时器中断的函数
    ffffffff98401c4f apic_timer_interrupt
    # 对称多处理（SMP）环境下的定时器中断处理
    ffffffff984026ba smp_apic_timer_interrupt
    ffffffff97b7e0f0 hrtimer_interrupt
    ffffffff97b7d920 __hrtimer_run_queues
    ffffffff97b7d6ee hrtimer_wakeup
    # 尝试唤醒一个进程的函数，是唤醒操作的上层逻辑
    ffffffff97b1d186 try_to_wake_up
    # 最终负责实际唤醒工作的函数
    ffffffff97b1c106 ttwu_do_wakeup
    # wakeuptime 工具基于 eBPF 技术实现，这是其中跟踪相关的函数
    ffffffff97c0dde2 bpf_trace_run1
    ffffffffc03a4aa6 exit_misc_binfmt
    # 获取 eBPF 堆栈 ID 的函数，用于获取当前堆栈的标识信息，以便进行跟踪和分析
    ffffffff97c0fe0e bpf_get_stackid_raw_tp
    # 唤醒者，即 swapper/1 唤醒了 mysqld
    waker:           swapper/1
    # waker 调用栈经过的时间，微秒（us）
        67523756

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# 注意这里的堆栈是用bcc libbpf-tools工具采集的，libbpf-tools/wakeuptime
Tracing blocked time (us) by kernel stack
    
    # 被唤醒者
    target:          mysqld
    # 从下往上，是waker（即swapper/9）经过的堆栈。堆栈方向和bcc tools处理后的相反。
    ffffffffc03c1d55 __this_module+0x3c55
    ffffffff97c0fe0e bpf_get_stackid_raw_tp+0x4e
    ffffffffc03c1d55 __this_module+0x3c55
    ffffffff97c0dde2 bpf_trace_run1+0x32
    ffffffff97b1c106 ttwu_do_wakeup+0x106
    # 内核中尝试唤醒指定进程（线程）的核心函数
    ffffffff97b1d186 try_to_wake_up+0x1a6
    # 高精度定时器相关的唤醒函数，当定时器到期时会触发这个函数
    ffffffff97b7d6ee hrtimer_wakeup+0x1e
    ffffffff97b7d920 __hrtimer_run_queues+0x100
    ffffffff97b7e0f0 hrtimer_interrupt+0x100
    ffffffff984026ba smp_apic_timer_interrupt+0x6a
    # 定时器中断函数被触发（经过一系列调用，最终调到上面的 try_to_wake_up ）
    ffffffff98401c4f apic_timer_interrupt+0xf
    ffffffff981435fb cpuidle_enter_state+0xdb
    ffffffff9814393c cpuidle_enter+0x2c
    ffffffff97b22a84 do_idle+0x234
    ffffffff97b22c7f cpu_startup_entry+0x6f
    ffffffff97a5929b start_secondary+0x19b
    # 辅助 CPU 启动的函数
    # 堆栈自底向上
    ffffffff97a00107 secondary_startup_64_no_verify+0xc2
              # swapper/9进程将 mysqld 唤醒的
              waker: swapper/9
    # 自底向上查看，唤醒者唤醒路径要花费的时间，us
    6006204   

1
2
3
4
# bcc libbpf-tools
# /home/workspace/bcc/libbpf-tools/wakeuptime
wakeuptime > out.stacks
stackcollapse.pl < out.stacks | flamegraph.pl --color=wakeup --title="Wakeup Time Flame Graph" --countname=us > wakeuptime_libpf_stress.svg

1
2
/usr/share/bcc/tools/offwaketime -f 20 > out.stacks
flamegraph.pl --color=chain --title="Off-Wake Time Flame Graph" --countname=us < out.stacks > offwaketime_out.svg

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# bcc tools采集的未折叠堆栈
[CentOS-root@xdlinux ➜ chain git:(main) ✗ ]$ cat not_folded_out.stacks 
Tracing blocked time (us) by user + kernel off-CPU and waker stack... Hit Ctrl-C to end.

    # 唤醒者
    waker:           swapper/5 0
    b'secondary_startup_64_no_verify'
    b'start_secondary'
    b'cpu_startup_entry'
    b'do_idle'
    b'cpuidle_enter'
    b'cpuidle_enter_state'
    # 定时器中断触发
    b'apic_timer_interrupt'
    b'smp_apic_timer_interrupt'
    # 定时器到期会触发这些函数。高精度定时器（hrtimer）相关函数
    b'hrtimer_interrupt'
    b'__hrtimer_run_queues'
    # 最终通过 hrtimer_wakeup 完成唤醒操作
    b'hrtimer_wakeup'
    --               --
    b'finish_task_switch'
    b'__sched_text_start'
    # schedule里往上调用__sched_text_start
    b'schedule'
    # I/O 操作等待
    b'read_events'
    b'do_io_getevents'
    b'__x64_sys_io_getevents'
    b'do_syscall_64'
    b'entry_SYSCALL_64_after_hwframe'
    # mysqld进行系统调用，然后依次往上执行堆栈，即体现的是被唤醒之前的堆栈/如何进入到阻塞状态的堆栈
    b'syscall'
    # 被唤醒者
    target:          mysqld 1543
        500247

1
2
3
4
5
6
# 下面命令以自己的工具路径示例，火焰图路径export PATH
export PATH=$PATH:/home/workspace/prog-playground/flamegraph_sample/tools/BPF-tools/old/2017-12-23/
export PATH=$PATH:/home/workspace/prog-playground/flamegraph_sample/tools/FlameGraph
# 工具执行有点小语法错误，按提示修改即可
fileiostacks.py -f 5 > out.stacks
flamegraph.pl --color=io --title="File I/O Time Flame Graph" --countname=us < out.stacks > fileio_out.svg

1
2
3
4
5
6
7
8
9
sysbench /usr/share/sysbench/oltp_read_write.lua \
    --mysql-host=localhost \
    --mysql-port=3306 \
    --mysql-user=root \
    --mysql-password=test \
    --mysql-db=xdtestdb \
    --tables=4 \
    --table-size=100 \
    prepare