Skip to content

09.xx Performance Monitoring

|525

Context Switching

Histograms

Sub-second offset Heat Map

Flame Graphs

Tools Ecosystem - perf - eBPF tools - perf-tools - bcc - bpftrace - GitHub - Netflix/flamescope: FlameScope is a visualization tool for exploring different time ranges as Flame Graphs. - Sleep(1) vs High Performance Times. Horizontal Line vs Diagonal Line

CPU Performance - PMC - Performance Monitoring Counters; 100+ PMCs. - One area is absence of PMC on cloud instances. - EC2 m4.16xL xen machines expose 7-9 PMCs - Use PMCs to correctly identify - Idle, Running and (new) Stalled! - Stalled is a problem which has cropped up in latest processors as they are starved on Memory - IPC - Intstructions per cycle - We can measure IPC using PMC - IPC < 0.2 bad. means we are stalled on memory. - IPC > 2.0 good.

Networking Issue

  1. bcc/tcplife

Disk Performance Issues.

  1. iostat
  2. iolatency |500

Typical Histogram latency for a 7200rpm HDD + 0 - 1ms -> Cache Hits + 8 - 16ms -> Cache Miss

  1. iosnoop and newer biosnoop the eBPF-io-snoop |525