Skip to content

gh-152769: Enable perf trampoline on Alpine#152774

Open
weixlu wants to merge 2 commits into
python:mainfrom
weixlu:perf
Open

gh-152769: Enable perf trampoline on Alpine#152774
weixlu wants to merge 2 commits into
python:mainfrom
weixlu:perf

Conversation

@weixlu

@weixlu weixlu commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Proposal of change

As mentioned in #152769, perf can‘t collect Python function-level hotspot info on Alpine Linux because perf trampoline support is not enabled there.

The trampoline is a CPU-architecture-specific feature and has little to do with OS or libc. Therefore, we can simply modify configure to reuse the same trampoline implementation as glibc.

Test results

I ran the tests on the following three platforms:

  • Alpine 3.22.4, musl 1.2.5, x86_64 (matching the buildbot env)
  • Alpine edge, musl 1.2.6, x86_64 (latest version)
  • Alpine 3.22.4, musl 1.2.5, aarch64 (specifically ARMv9 Neoverse N2)

The three profiling-related tests now pass on all of these platforms:

test_perf_profiler
test_perfmaps
test_samply_profiler

Before this PR, these tests were skipped.

I also manually collected profiling data with perf and confirmed that Python-level functions appear in the profiling output.

How to reproduce

  1. When running Alpine in a container, note that the container needs to be started with the additional --privileged option.

  2. Apart from build dependencies, install perf and samply. samply needs to be built with Rust:

apk add perf rust cargo
export PATH="$HOME/.cargo/bin:$PATH"
cargo install --locked samply
  1. Run the tests:
./python -m test test_perf_profiler test_perfmaps test_samply_profiler
  1. To manually check profiling data, refer to the scripts described in Lack of perf support on Alpine Linux #152769.

@weixlu

weixlu commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

@vstinner Hi, this is the PR fixing perf trampolines on musl. Could you please take a look?

@thesamesam

thesamesam commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

See also #122370 (and #121608).

@weixlu

weixlu commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for pointing out, @thesamesam — you're right. I'll close #152769 as a duplicate of #121608 so the discussion stays in one place.

Hi, @corona10, since you authored the original #122370, I'd like to defer to you on how best to handle this.

  • Back in 2024 when you opened the PR, the profiling tests were skipped because the Alpine perf was too old.
  • Now with the latest version of Alpine, that's no longer an issue — I verified on three platforms and test_perf_profiler / test_perfmaps / test_samply_profiler now pass.
  • The configure logic has also changed a bit in the meantime, so the original diff in gh-121608: Add supporting perf trampoline on -linux-musl #122370 doesn't apply cleanly to main anymore. This PR is the same change on the current code.

Whatever works best for you — if you'd prefer to update and land #122370 yourself, I'm happy to close mine; or feel free to use this PR if that's easier. Thanks!

@vstinner

vstinner commented Jul 9, 2026

Copy link
Copy Markdown
Member

I have issues to try samply in my Alpine Linux (podman) container running on Fedora 44 :-(

First, it complained because of perf_event_paranoid:

# samply record --save-only --output profile.json.gz /root/cpython/python -c 'print("hello")' 

'/proc/sys/kernel/perf_event_paranoid' is currently set to 2.
In order for samply to work with a non-root user, this level needs
to be set to 1 or lower.
You can execute the following command and then try again:
    echo '1' | sudo tee /proc/sys/kernel/perf_event_paranoid

I ran echo '1' | sudo tee /proc/sys/kernel/perf_event_paranoid, but then it fails with:

# samply record --save-only --output profile.json.gz /root/cpython/python -c 'print("hello")' |cat
Failed to start profiling: Operation not permitted (os error 1)

That's maybe because I'm running Alpine Linux in a podman container which is running as root?

Second, test_samply_profiler hangs randomly on samply_command_works(). I can reproduce the issue in the command line, without Python:

# samply record --save-only --output profile.json.gz /root/cpython/python -c 'print("hello")' |cat
Failed to start profiling: Operation not permitted (os error 1)

The command hangs. If I run samply in strace, it no longer hangs, so I cannot debug the issue :-(

# strace -o trace samply record --save-only --output profile.json.gz /root/cpython/python -c 'print("hello")' |cat
Failed to start profiling: Operation not permitted (os error 1)

By the way, test_samply_profiler runs samply_command_works() 3 times, which is inefficient! I wrote #153424 to only run samply_command_works() once.

@vstinner

vstinner commented Jul 9, 2026

Copy link
Copy Markdown
Member

I also had permission issues with running Linux perf in my Alpine Linux container:

# perf record pwd
perf_event_open(..., PERF_FLAG_FD_CLOEXEC) failed with unexpected error 1 (Operation not permitted)
perf_event_open(..., 0) failed unexpectedly with error 1 (Operation not permitted)
Missing support for build id in kernel mmap events.
Disable this warning with --no-buildid-mmap
Error:
Failure to open event 'cpu/cycles/P' on PMU 'cpu' which will be removed.
No fallback found for 'cpu/cycles/P' for error 1
Error:
Failure to open any events for recording.

Note: I tried setting /proc/sys/kernel/perf_event_paranoid to -1, but it didn't help.

@vstinner

vstinner commented Jul 9, 2026

Copy link
Copy Markdown
Member

@corona10: Are you able to run the 3 tests on Alpine Linux successfully?

@weixlu

weixlu commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

@vstinner Hi, looks like it's related to permission issues. When you start your podman, did you add the flag --privileged? like podman run -it --privileged alpine:edge sh

@vstinner

vstinner commented Jul 9, 2026

Copy link
Copy Markdown
Member

Ah, I forgot to mention that I tried running podman run --privileged, but it didn't solve the permission issue.

@weixlu

weixlu commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Could you let me know which version of the alpine image you're using? I'll see if I can reproduce it.

@weixlu

weixlu commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

And perf_event_paranoid is 1 here on my side. I'm also running the container as root

~ # cat /proc/sys/kernel/perf_event_paranoid
1
~ # perf record pwd
/root
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.017 MB perf.data ]
~ # perf version
perf version 6.15.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants