unwind: target platform=x86 is not supported
Sunday, 1 October 2017
I upgraded Debian on my second workstation specifically so that the perf tool would gain the -k option.
-k specifies the clock used by perf. perf is a Linux performance monitoring tool with many features - in this case, I am using it to capture a trace of system events while a program runs. Each event has a timestamp, and the timestamp comes from the clock specified by -k.
You may not be aware that the Linux kernel provides multiple clocks as well as the "obvious" clock which gives the time of day. For instance, there is CLOCK_BOOTTIME, recording the time that has elapsed since boot. There is also CLOCK_MONOTONIC, which is a clock that is guaranteed to never run backwards. (The time of day may jump backwards, e.g. due to daylight savings time or an adjustment to match a time server, but the monotonic clock will always run forwards.) All of these clocks are subtly different, and if you need to synchronise a "perf" trace with a trace made somewhere else (e.g. within a program) then you need to be specific about which clock you want.
perf is tied to the Linux kernel version, so upgrading perf means upgrading the kernel too. Though I could upgrade the kernel "by hand", I don't really want to deal with the hassle of possible conflicts between user-space programs and a new kernel version, so I decided to just upgrade the Debian distribution from 8.x to 9.1. The upgrade also updates the kernel version from 3.16 to 4.9, and perf along with it.
But I found that the new perf was unusable, because "unwind: target platform=x86 is not supported". I solved it by rebuilding "perf", and I'll explain how shortly.
The error appears when you run "perf script" to convert binary trace data ("perf.data") to a form that's easily processed by other tools. Previously I've written about flame graphs and how they're a useful way to visualise a profile, and on Linux a flame graph is normally generated via "perf script". In this case, I am using traces instead because I am investigating how multiple threads interact and block each other, and profiles are not useful for that. But perf is a general tracing tool, and profiling is just a small part of what it can do. It can trace system calls and many other sorts of event. In this case I was using the following "perf record" command line:
perf record \
-k CLOCK_MONOTONIC \
-e 'ext4_sync_file_enter,ext4:ext4_sync_file_exit' \
-a -g \
program
This records sync events (at the filesystem level - I can't record the "fsync" system call because that only shows up x64 system calls, and my program is x86). I'm using CLOCK_MONOTONIC in order to synchronise with other traces being recorded within my program (it's a work thing, so I can't go into detail about what it actually does). I record all system events (-a) and I record backtraces for every event (-g).
But because my program is x86, the new "perf" cannot decode the backtraces to symbols, so I get output like this:
program 31317 1032829.601824: cycles:ppp:
56589566 [unknown] ([unknown])
565895b0 [unknown] ([unknown])
56589585 [unknown] ([unknown])
565895b0 [unknown] ([unknown])
56589606 [unknown] ([unknown])
f7525276 [unknown] ([unknown])
These 8-digit hex numbers are code addresses in my program, and normally they'd be translated to symbols such as "main", but that's not happening. The "unwind: target platform=x86 is not supported" warning is telling me why. perf doesn't have support for understanding x86 ELF binaries. This is a pain.
In order to fully support x86 and x64, perf must be linked against a "libunwind" that supports both x86 and x64, and you don't get that on Debian now. There is a conflict between the x64 version of libunwind and the x86 version:
# apt-get install libunwind-dev:i386 libunwind-dev:amd64
...
The following packages have unmet dependencies:
libunwind-dev : Conflicts: libunwind-dev:i386 but 1.1-4.1 is to be installed
libunwind-dev:i386 : Depends: libunwind8:i386 (= 1.1-4.1) but it is not going to be installed
Conflicts: libunwind-dev but 1.1-4.1 is to be installed
E: Unable to correct problems, you have held broken packages.
I struggle to understand the cryptic messages produced by "apt", indeed I have always done so, ever since the first time I used Linux when there was only "dpkg", but I think the conflict is because both packages contain different copies of the same file. It is "/usr/include/libunwind.h". So the packages can't be installed together. As a result, "perf" has only been built against one of them, and it's the x64 version because this is an x64 distribution. However, that's no help for people still working on x86 programs.
I felt the easiest solution was to rebuild "libunwind" and "perf" by hand, enabling the appropriate support. It wasn't straightforward. Here's how I did it.
- I got the Linux kernel source matching the version shipped with Debian 9.1. This was 4.9.30. perf is part of the kernel source (find it in tools/perf). I think it's best to get this from Debian rather than anywhere else because this is the best way to be sure you have exactly the right version.
- I also got the libunwind source for the version shipped by Debian. This is not the most recent version of the library, but now is not the time to attempt to upgrade that.
- I extracted the downloaded source code and copied the Linux kernel configuration from /boot/config-4.9.0-3-amd64 to .config in the kernel directory. (With appropriate build tools ("apt-get install build-essential") it was immediately possible to build perf, but the compiled version had the same problems as the original, namely that libunwind didn't support x86 symbols.)
- I built libunwind for x86:
tar xvzf libunwind_1.1.orig.tar.gz cd libunwind-1.1/ ./configure --prefix=$PWD/install --target=i686-linux make make install - I copied two files from libunwind-x86 to the Linux source code directory.
- include/libunwind-x86.h is copied to linux-4.9.30/tools/include/
- lib/libunwind-x86.a is copied to linux-4.9.30/tools/perf/
- Now I found I had to hack the build configuration used for perf to make it
use this version of libunwind in addition to the x64 version. Here's how.
I got perf's build system to dump the build configuration:
cd linux-4.9.30/tools/perf make -f Makefile.perf feature-dump cp FEATURE-DUMP FEATURE-DUMP-MODIFIED - Then I edited "FEATURE-DUMP-MODIFIED" to set "feature-libunwind-x86=1".
- Finally I built perf with the modified configuration and new perf:
cd linux-4.9.30/tools/perf make -f Makefile.perf \ FEATURES_DUMP=$PWD/FEATURE-DUMP-MODIFIED LDFLAGS=-L$PWD
This completed the process, giving me a version of perf that was capable of decoding x86 backtraces as well as x64 backtraces. The binary is linux-4.9.30/tools/perf/perf. It is dynamically linked against x64 libunwind, and statically linked against x86 libunwind. I can rerun "perf script" and see the symbls in the program:
program 31317 1032829.601824: cycles:ppp:
566 example2+0xffffffff534ee006 (/tmp/program)
5b0 example1+0xffffffff534ee025 (/tmp/program)
585 example2+0xffffffff534ee025 (/tmp/program)
5b0 example1+0xffffffff534ee025 (/tmp/program)
606 main+0xffffffff534ee050 (/tmp/program)
18276 __libc_start_main+0xfffffffe115e60f6 (...)
One annoyance with this new version is that it can't be easily installed elsewhere (e.g. in my personal "bin" directory) because when invoked without an absolute path, "perf" acts as a frontend to the system "perf", helpfully launching the right version for the current kernel. Unfortunately this will invoke the "perf" that lacks the x86 feature. So I have to explicitly invoke "perf" with an absolute path every time. However, "perf" is normally run from shell scripts anyway, since it is normally preceded by some experimental setup, and followed by steps to convert results into a human-readable form, so it is not a big deal to have long command lines.
Here are all of the steps I used, in the form of a shell script:
#!/bin/bash -xe
cd $HOME
mkdir perf
cd perf
cp /c/temp/linux_4.9.30.orig.tar.xz .
cp /c/temp/libunwind_1.1.orig.tar.gz .
tar xf linux_4.9.30.orig.tar.xz
tar xf libunwind_1.1.orig.tar.gz
cd linux-4.9.30/
cp /boot/config-4.9.0-3-amd64 .config
cd ../libunwind-1.1/
./configure --prefix=$PWD/install --target=i686-linux
make
make install
cp install/include/libunwind-x86.h ../linux-4.9.30/tools/include/
cp install/lib/libunwind-x86.a ../linux-4.9.30/tools/perf/
cd ../linux-4.9.30/tools/perf/
make -f Makefile.perf feature-dump
cp FEATURE-DUMP FEATURE-DUMP-MODIFIED
sed -i 's/\(feature-libunwind-x86\).*/\1=1/' FEATURE-DUMP-MODIFIED
make -f Makefile.perf \
FEATURES_DUMP=$PWD/FEATURE-DUMP-MODIFIED LDFLAGS=-L$PWD