Demonstrations of compactstall, the Linux eBPF/bcc version.

compactsnoop traces the compact zone system-wide, and print various details.

Example output (manual trigger by echo 1 > /proc/sys/vm/compact_memory):

# ./compactsnoop

COMM PID NODE ZONE ORDER MODE LAT(ms) STATUS

zsh 23685 0 ZONE_DMA -1 SYNC 0.025 complete

zsh 23685 0 ZONE_DMA32 -1 SYNC 3.925 complete

zsh 23685 0 ZONE_NORMAL -1 SYNC 113.975 complete

zsh 23685 1 ZONE_NORMAL -1 SYNC 81.57 complete

zsh 23685 0 ZONE_DMA -1 SYNC 0.02 complete

zsh 23685 0 ZONE_DMA32 -1 SYNC 4.631 complete

zsh 23685 0 ZONE_NORMAL -1 SYNC 113.975 complete

zsh 23685 1 ZONE_NORMAL -1 SYNC 80.647 complete

zsh 23685 0 ZONE_DMA -1 SYNC 0.020 complete

zsh 23685 0 ZONE_DMA32 -1 SYNC 3.367 complete

zsh 23685 0 ZONE_NORMAL -1 SYNC 115.18 complete

zsh 23685 1 ZONE_NORMAL -1 SYNC 81.766 complete

zsh 23685 0 ZONE_DMA -1 SYNC 0.025 complete

zsh 23685 0 ZONE_DMA32 -1 SYNC 4.346 complete

zsh 23685 0 ZONE_NORMAL -1 SYNC 114.570 complete

zsh 23685 1 ZONE_NORMAL -1 SYNC 80.820 complete

zsh 23685 0 ZONE_DMA -1 SYNC 0.026 complete

zsh 23685 0 ZONE_DMA32 -1 SYNC 4.611 complete

zsh 23685 0 ZONE_NORMAL -1 SYNC 113.993 complete

zsh 23685 1 ZONE_NORMAL -1 SYNC 80.928 complete

zsh 23685 0 ZONE_DMA -1 SYNC 0.02 complete

zsh 23685 0 ZONE_DMA32 -1 SYNC 3.889 complete

zsh 23685 0 ZONE_NORMAL -1 SYNC 113.776 complete

zsh 23685 1 ZONE_NORMAL -1 SYNC 80.727 complete

^C

While tracing, the processes alloc pages due to memory fragmentation is too

serious to meet contiguous memory requirements in the system, compact zone

events happened, which will increase the waiting delay of the processes.

compactsnoop can be useful for discovering when compact_stall(/proc/vmstat)

continues to increase, whether it is caused by some critical processes or not.

The STATUS include (CentOS 7.6's kernel)

compact_status = {

# COMPACT_SKIPPED: compaction didn't start as it was not possible or direct reclaim was more suitable

0: "skipped",

# COMPACT_CONTINUE: compaction should continue to another pageblock

1: "continue",

# COMPACT_PARTIAL: direct compaction partially compacted a zone and there are suitable pages

2: "partial",

# COMPACT_COMPLETE: The full zone was compacted

3: "complete",

}

or (kernel 4.7 and above)

compact_status = {

# COMPACT_NOT_SUITABLE_ZONE: For more detailed tracepoint output - internal to compaction

0: "not_suitable_zone",

# COMPACT_SKIPPED: compaction didn't start as it was not possible or direct reclaim was more suitable

1: "skipped",

# COMPACT_DEFERRED: compaction didn't start as it was deferred due to past failures

2: "deferred",

# COMPACT_NOT_SUITABLE_PAGE: For more detailed tracepoint output - internal to compaction

3: "no_suitable_page",

# COMPACT_CONTINUE: compaction should continue to another pageblock

4: "continue",

# COMPACT_COMPLETE: The full zone was compacted scanned but wasn't successful to compact suitable pages.

5: "complete",

# COMPACT_PARTIAL_SKIPPED: direct compaction has scanned part of the zone but wasn't successful to compact suitable pages.

6: "partial_skipped",

# COMPACT_CONTENDED: compaction terminated prematurely due to lock contentions

7: "contended",

# COMPACT_SUCCESS: direct compaction terminated after concluding that the allocation should now succeed

8: "success",

}

The -p option can be used to filter on a PID, which is filtered in-kernel. Here

I've used it with -T to print timestamps:

# ./compactsnoop -Tp 24376

TIME(s) COMM PID NODE ZONE ORDER MODE LAT(ms) STATUS

101.364115000 zsh 24376 0 ZONE_DMA -1 SYNC 0.025 complete

101.364555000 zsh 24376 0 ZONE_DMA32 -1 SYNC 3.925 complete

^C

This shows the zsh process allocs pages, and compact zone events happening,

and the delays are not affected much.

A maximum tracing duration can be set with the -d option. For example, to trace

for 2 seconds:

# ./compactsnoop -d 2

COMM PID NODE ZONE ORDER MODE LAT(ms) STATUS

zsh 26385 0 ZONE_DMA -1 SYNC 0.025444 complete

^C

The -e option prints out extra columns

# ./compactsnoop -e

COMM PID NODE ZONE ORDER MODE FRAGIDX MIN LOW HIGH FREE LAT(ms) STATUS

summ 28276 1 ZONE_NORMAL 3 ASYNC 0.728 11284 14105 16926 14193 3.58 partial

summ 28276 0 ZONE_NORMAL 2 ASYNC -1.000 11043 13803 16564 14479 0.0 complete

summ 28276 1 ZONE_NORMAL 2 ASYNC -1.000 11284 14105 16926 14785 0.019 complete

summ 28276 0 ZONE_NORMAL 2 ASYNC -1.000 11043 13803 16564 15199 0.006 partial

summ 28276 1 ZONE_NORMAL 2 ASYNC -1.000 11284 14105 16926 17360 0.030 complete

summ 28276 0 ZONE_NORMAL 2 ASYNC -1.000 11043 13803 16564 15443 0.024 complete

summ 28276 1 ZONE_NORMAL 2 ASYNC -1.000 11284 14105 16926 15634 0.018 complete

summ 28276 1 ZONE_NORMAL 3 ASYNC 0.832 11284 14105 16926 15301 0.006 partial

summ 28276 0 ZONE_NORMAL 2 ASYNC -1.000 11043 13803 16564 14774 0.005 partial

summ 28276 1 ZONE_NORMAL 3 ASYNC 0.733 11284 14105 16926 19888 0.012 partial

^C

The FRAGIDX is short for fragmentation index, which only makes sense if an

allocation of a requested size would fail. If that is true, the fragmentation

index indicates whether external fragmentation or a lack of memory was the

problem. The value can be used to determine if page reclaim or compaction

should be used.

Index is between 0 and 1 so return within 3 decimal places

0 => allocation would fail due to lack of memory

1 => allocation would fail due to fragmentation

We can see the whole buddy's fragmentation index from /sys/kernel/debug/extfrag/extfrag_index

The MIN/LOW/HIGH shows the watermarks of the zone, which can also get from

/proc/zoneinfo, and FREE means nr_free_pages (can be found in /proc/zoneinfo too).

The -K option prints out kernel stack

# ./compactsnoop -K -e

summ 28276 0 ZONE_NORMAL 3 ASYNC 0.528 11043 13803 16564 22654 13.258 partial

kretprobe_trampoline+0x0

try_to_compact_pages+0x121

__alloc_pages_direct_compact+0xac

__alloc_pages_slowpath+0x3e9

__alloc_pages_nodemask+0x404

alloc_pages_current+0x98

new_slab+0x2c5

___slab_alloc+0x3ac

__slab_alloc+0x40

kmem_cache_alloc_node+0x8b

copy_process+0x18e

do_fork+0x91

sys_clone+0x16

stub_clone+0x44

summ 28276 1 ZONE_NORMAL 3 ASYNC -1.000 11284 14105 16926 22074 0.008 partial

kretprobe_trampoline+0x0

try_to_compact_pages+0x121

__alloc_pages_direct_compact+0xac

__alloc_pages_slowpath+0x3e9

__alloc_pages_nodemask+0x404

alloc_pages_current+0x98

new_slab+0x2c5

___slab_alloc+0x3ac

__slab_alloc+0x40

kmem_cache_alloc_node+0x8b

copy_process+0x18e

do_fork+0x91

sys_clone+0x16

stub_clone+0x44

summ 28276 0 ZONE_NORMAL 3 ASYNC 0.527 11043 13803 16564 25653 9.812 partial

kretprobe_trampoline+0x0

try_to_compact_pages+0x121

__alloc_pages_direct_compact+0xac

__alloc_pages_slowpath+0x3e9

__alloc_pages_nodemask+0x404

alloc_pages_current+0x98

new_slab+0x2c5

___slab_alloc+0x3ac

__slab_alloc+0x40

kmem_cache_alloc_node+0x8b

copy_process+0x18e

do_fork+0x91

sys_clone+0x16

stub_clone+0x44

# ./compactsnoop -h

usage: compactsnoop.py [-h] [-T] [-p PID] [-d DURATION] [-K] [-e]

Trace compact zone

optional arguments:

-h, --help show this help message and exit

-T, --timestamp include timestamp on output

-p PID, --pid PID trace this PID only

-d DURATION, --duration DURATION

total duration of trace in seconds

-K, --kernel-stack output kernel stack trace

-e, --extended_fields

show system memory state

examples:

./compactsnoop # trace all compact stall

./compactsnoop -T # include timestamps

./compactsnoop -d 10 # trace for 10 seconds only

./compactsnoop -K # output kernel stack trace

./compactsnoop -e # show extended fields