qemu-user-static - part 2 register
qemu-user-static
enables us to run multi architecture environment such as ARM on x86_64 like this.
$ docker run --rm --privileged multiarch/qemu-user-static:register --reset $ docker run --rm -t multiarch/ubuntu-debootstrap:arm64-bionic uname -a Linux 28c784e9c7bc 4.4.0-101-generic #124~14.04.1-Ubuntu SMP Fri Nov 10 19:05:36 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux
On the part 1, we leaned the qemu-user-static
is to create the small container including QEMU binary (qemu-foo-static
) or the tar.gz file then upload it to the DockerHub container registry.
The part 2, we dig how the register container works reading the source code.
What does the register work?
First, register/Dockerfile
file on [1] is the top level file.
busybox
is small Linux distribution.
FROM busybox ENV QEMU_BIN_DIR=/usr/bin ADD ./register.sh /register ADD https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh RUN chmod +x /qemu-binfmt-conf.sh ENTRYPOINT ["/register"]
What is https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh
?
Download it to check it.
$ wget https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh
It is a script managed at QEMU GitHub repository [2] https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh
Next ENTRYPOINT ["/register"]
is called as the entry point. In above case, register.sh --reset
is called.
The steps in the file are
- Check if
/proc/sys/fs/binfmt_misc
directory exists mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
if/proc/sys/fs/binfmt_misc/register
files does not existfind /proc/sys/fs/binfmt_misc -type f -name 'qemu-*' -exec sh -c 'echo -1 > {}' \;
if--reset
option is specified.exec /qemu-binfmt-conf.sh --qemu-suffix "-static" --qemu-path="${QEMU_BIN_DIR}" $@
Check if /proc/sys/fs/binfmt_misc
directory exists
When I checked /proc/sys/fs/binfmt_misc/
directory on my host os (Fedora), below is the result.
$ ls -l /proc/sys/fs/binfmt_misc/ total 0 -rw-r--r-- 1 root root 0 Apr 7 14:55 qemu-aarch64 -rw-r--r-- 1 root root 0 Apr 7 14:55 qemu-aarch64_be ... -rw-r--r-- 1 root root 0 Apr 7 14:55 qemu-xtensaeb --w------- 1 root root 0 Apr 7 14:55 register -rw-r--r-- 1 root root 0 Apr 7 14:55 status
So, step 2. mount binfmt_misc
and 3. find ..
is to create zero byte files /proc/sys/fs/binfmt_misc/qemu-*
.
What is "binfmt_misc"? I found an article [3] [4]
According to [3]
The binfmt-support package contains a helper script to easily register/unregister binary formats with the kernel using the binfmt_misc module.
Install qemu, binfmt-support, and qemu-user-static:
apt-get install qemu binfmt-support qemu-user-static
There is a sub deb package of qemu deb package. https://packages.debian.org/sid/qemu-user-static
According to [4]
5.3.9.2. /proc/sys/fs/ This directory contains an array of options and information concerning various aspects of the file system, including quota, file handle, inode, and dentry information. The binfmt_misc/ directory is used to provide kernel support for miscellaneous binary formats.
exec /qemu-binfmt-conf.sh --qemu-suffix "-static" --qemu-path="${QEMU_BIN_DIR}" $@
Below is the actual command to be executed.
./qemu-binfmt-conf.sh --qemu-suffix "-static" --qemu-path="/usr/bin" --reset
Build the register container by myself to check the qemu-binfmt-conf.sh
's behavior.
qemu-binfmt-conf.sh
was downloaded above.
Add debug set -x
.
$ vi register/qemu-binfmt-conf.sh ... set -x ... $ vi register/register.sh $ git diff register/register.sh set -x QEMU_BIN_DIR=${QEMU_BIN_DIR:-/usr/bin} $ vi register/Dockerfile ... # ADD https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh <= Comment out this line. ADD ./qemu-binfmt-conf.sh /qemu-binfmt-conf.sh <= Add this line. RUN chmod +x /qemu-binfmt-conf.sh ENTRYPOINT ["/register"] $ docker build --rm -t junaruga/qemu-user-static:register register ... Successfully tagged junaruga/qemu-user-static:register $ docker image ls -a | grep junaruga/qemu-user-static junaruga/qemu-user-static register 50384f9a6262 9 seconds ago 1.23MB $ docker run --rm --privileged junaruga/qemu-user-static:register --reset
Here is the output log.
Seeing qemu-binfmt-conf.sh
buttom lines, the 2 main steps (functions) are below.
qemu_check_bintfmt_misc qemu_set_binfmts
qemu_check_bintfmt_misc
is to check valid system system status.
qemu_check_bintfmt_misc() { # load the binfmt_misc module if [ ! -d /proc/sys/fs/binfmt_misc ]; then if ! /sbin/modprobe binfmt_misc ; then exit 1 fi fi if [ ! -f /proc/sys/fs/binfmt_misc/register ]; then if ! mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc ; then exit 1 fi fi qemu_check_access /proc/sys/fs/binfmt_misc/register }
qemu_target_list
is for ${qemu_target_list}
, run $BINFMT_SET
(= qemu_register_interpreter
function).
qemu_target_list="i386 i486 alpha arm armeb sparc32plus ppc ppc64 ppc64le m68k \ mips mipsel mipsn32 mipsn32el mips64 mips64el \ sh4 sh4eb s390x aarch64 aarch64_be hppa riscv32 riscv64 xtensa xtensaeb \ microblaze microblazeel or1k x86_64"
qemu_set_binfmts() { # probe cpu type host_family=$(qemu_get_family) # register the interpreter for each cpu except for the native one for cpu in ${qemu_target_list} ; do magic=$(eval echo \$${cpu}_magic) mask=$(eval echo \$${cpu}_mask) family=$(eval echo \$${cpu}_family) if [ "$magic" = "" ] || [ "$mask" = "" ] || [ "$family" = "" ] ; then echo "INTERNAL ERROR: unknown cpu $cpu" 1>&2 continue fi qemu="$QEMU_PATH/qemu-$cpu" if [ "$cpu" = "i486" ] ; then qemu="$QEMU_PATH/qemu-i386" fi qemu="$qemu$QEMU_SUFFIX" if [ "$host_family" != "$family" ] ; then $BINFMT_SET fi done }
qemu_register_interpreter
registers a content to /proc/sys/fs/binfmt_misc/register
procfs file.
When a stdout content redirected to the file /proc/sys/fs/binfmt_misc/register
.
The bahabior of >
is not like normal file.
qemu_register_interpreter() { echo "Setting $qemu as binfmt interpreter for $cpu" qemu_generate_register > /proc/sys/fs/binfmt_misc/register }
qemu_generate_register() { flags="" if [ "$CREDENTIAL" = "yes" ] ; then flags="OC" fi if [ "$PERSISTENT" = "yes" ] ; then flags="${flags}F" fi echo ":qemu-$cpu:M::$magic:$mask:$qemu:$flags" }
https://en.wikipedia.org/wiki/Binfmt_misc
The executable formats are registered through the special purpose file system binfmt_misc file-system interface (usually mounted under part of /proc). This is either done directly by sending special sequences to the register procfs file or using a wrapper like Debian-based distributions binfmt-support package[3] or systemd's systemd-binfmt.service[4][5].
Below code for each target cpu is what the register is doing.
echo ":qemu-$cpu:M::$magic:$mask:$qemu:$flags" > /proc/sys/fs/binfmt_misc/register
On next blog, I dig the compatible images. [5]
References
- [1] qemu-user-static: https://github.com/multiarch/qemu-user-static
- [2] https://github.com/qemu/QEMU
- [3] QemuUserEmulation - Debian Wiki
- [4] 5.3.9.2. /proc/sys/fs/
- [5] compatible images: https://github.com/multiarch/qemu-user-static#compatible-images
qemu-user-static - part 1 summary
qemu-user-static [1] is a software to enable qemu [2] consumable to build and run multi architecture (ARM 64-bit, ARM 32-bit, Intel 32-bit and etc) on x86_64 Intel processor environment.
If you run docker or a docker compatible software, you can run commands like this.
$ docker run --rm --privileged multiarch/qemu-user-static:register --reset $ docker run --rm -t multiarch/ubuntu-debootstrap:arm64-bionic uname -a Linux 28c784e9c7bc 4.4.0-101-generic #124~14.04.1-Ubuntu SMP Fri Nov 10 19:05:36 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux
$ docker run --rm --privileged multiarch/qemu-user-static:register --reset $ docker run --rm -it multiarch/ubuntu-debootstrap:arm64-bionic bash root@1e773d5d74af:/# uname -a Linux 1e773d5d74af 5.0.5-200.fc29.x86_64 #1 SMP Wed Mar 27 20:58:04 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux root@1e773d5d74af:/# exit exit
Why is this software important now?
Because of high growth of IoT that is to use a kind of mobile embedded device. HPC (super comuputing) on not only Intel CPU (x86_64), but also on ARM 64-bit CPU (aarch64), ARM 32-bit, IBM ppc64le, and etc. Open source project could be bulit on the multi-architecture environments.
But everyone does not have a environment for ARM and IBM architecture.
Not only that, every CI (Continuous Integration) Service that is important for open source development does not support multi architecture.
qemu-user-static
enables users to run their software on the environments on local or CI "easily".
My motivaion to write this blog is that I want to spread this software to many people to promte softwares supporting the multi-architecture environments.
How does it work? (Reading the source code)
I did read the souce code to understand what it is doing.
Current latest version's commit hash is 20674ec
(version v3.1.0-3
+ 2 commits)
https://github.com/multiarch/qemu-user-static/commit/20674ec
$ git clone https://github.com/multiarch/qemu-user-static.git $ cd qemu-user-static $ git chckout 20674ec <= if you like reproducing below steps exactlly.
See .travis.yml
. It is a top level file to understand it.
At first, install necessary packages.
If you are using Debian base Linux like Ubuntu, like this.
$ sudo apt-get install jq rpm2cpio cpio
If you are using RPM base Linux such as Fedora and CentOS, like this.
$ sudo dnf install jq cpio
or
$ sudo yum install jq cpio
Then download qemu
RPM (pre-built binaries) file from Fedora Project.
You can see all the built files from here.
Right now f30 (Fedora 30) is the latest stable (beta) version.
$ wget --content-disposition https://kojipkgs.fedoraproject.org/packages/qemu/3.1.0/6.fc30/x86_64/qemu-user-static-3.1.0-6.fc30.x86_64.rpm
Extract downloaded RPM file.
$ rpm2cpio qemu-user-static-3.1.0-6.fc30.x86_64.rpm | cpio -dimv ./usr/bin/qemu-aarch64-static ./usr/bin/qemu-aarch64_be-static ./usr/bin/qemu-alpha-static ...
See extracted files under usr
.
$ ls usr/ bin/ lib/ share/ $ ls usr/bin/ qemu-aarch64_be-static qemu-mips64-static qemu-riscv64-static ...
Then remaining steps are as following.
generate_tarballs.sh
publish.sh
update.sh
docker push
generate_tarballs.sh
generate_tarballs.sh
copies ./usr/bin/qemu-*-static
in the above extracted files, and creates tar.gz files to releases
directory.
Run below command with a shell debug mode sh -x
to see each commands in the script file.
$ sh -x ./generate_tarballs.sh + rm -rf releases + mkdir -p releases + cp ./usr/bin/qemu-aarch64_be-static ./usr/bin/qemu-aarch64-static ... releases/ + cd releases/ + for file in * + tar -czf qemu-aarch64_be-static.tar.gz qemu-aarch64_be-static + cp qemu-aarch64_be-static.tar.gz x86_64_qemu-aarch64_be-static.tar.gz ... + tar -czf qemu-xtensa-static.tar.gz qemu-xtensa-static + cp qemu-xtensa-static.tar.gz x86_64_qemu-xtensa-static.tar.gz
See below files created under releases
directory.
qemu-foo-static.tar.gz
and x86_64_qemu-foo-static.tar.gz
is same content right now.
$ ls releases | grep arch64 qemu-aarch64_be-static qemu-aarch64_be-static.tar.gz qemu-aarch64-static qemu-aarch64-static.tar.gz x86_64_qemu-aarch64_be-static.tar.gz x86_64_qemu-aarch64-static.tar.gz
publish.sh
DO NOT RUN publish.sh
by yourself.
This script uploads all the file under the releases
directory to GitHub release page like https://github.com/multiarch/qemu-user-static/releases/tag/v3.1.0-3 .
Check the uploaded command version. It is QEMU 3.1.
$ wget https://github.com/multiarch/qemu-user-static/releases/download/v3.1.0-3/qemu-aarch64-static $ chmod +x qemu-aarch64-static $ ./qemu-aarch64-static --version qemu-aarch64 version 3.1.0 (qemu-3.1.0-6.fc30) Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers
update.sh
You can run update.sh
on your local.
This script creates container images. The each container images has /usr/bin/x86_64_qemu-foo-static.tar.gz
A content built by docker build
is small. Just adding the command on the scratch (no content) container.
FROM scratch ADD https://github.com/${REPO}/releases/download/v${VERSION}/${from_arch}_qemu-${to_arch}-static.tar.gz /usr/bin
Also the script creates a container by register/Dockerfile
.
busybox
is a very small sized Linux distribution often used a container.
register/Dockerfile
is like this.
About what the file is doing, I will dig it on next blog.
FROM busybox ENV QEMU_BIN_DIR=/usr/bin ADD ./register.sh /register ADD https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh RUN chmod +x /qemu-binfmt-conf.sh ENTRYPOINT ["/register"]
$ docker build -t ${REPO}:register register
So, actually build it on local.
$ sh -x ./update.sh -v "3.1.0-3" -r "multiarch/qemu-user-static" ... + from_arch=x86_64 + to_archs='aarch64 alpha arm armeb cris hppa i386 m68k microblaze microblazeel mips mips64 mips64el mipsel mipsn32 mipsn32el nios2 or1k ppc ppc64 ppc64abi32 ppc64le s390x sh4 sh4eb sparc sparc32plus sparc64 x86_64' ... + for to_arch in $to_archs + '[' x86_64 '!=' aarch64 ']' + docker build -t multiarch/qemu-user-static:x86_64-aarch64 - Sending build context to Docker daemon 2.048kB Step 1/2 : FROM scratch ---> Step 2/2 : ADD https://github.com/multiarch/qemu-user-static/releases/download/v3.1.0-3/x86_64_qemu-aarch64-static.tar.gz /usr/bin Downloading 1.61MB/1.61MB ---> 58b61c68fb02 Successfully built 58b61c68fb02 Successfully tagged multiarch/qemu-user-static:x86_64-aarch64 + docker tag multiarch/qemu-user-static:x86_64-aarch64 multiarch/qemu-user-static:aarch64 ... + docker build -t multiarch/qemu-user-static:register register Sending build context to Docker daemon 3.584kB Step 1/6 : FROM busybox latest: Pulling from library/busybox fc1a6b909f82: Pull complete Digest: sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd Status: Downloaded newer image for busybox:latest ---> af2f74c517aa Step 2/6 : ENV QEMU_BIN_DIR=/usr/bin ---> Running in 19568e18a181 Removing intermediate container 19568e18a181 ---> 9861f53c0ee0 Step 3/6 : ADD ./register.sh /register ---> cc357a1c64ec Step 4/6 : ADD https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh Downloading 13.24kB ---> 88c740e9bbe6 Step 5/6 : RUN chmod +x /qemu-binfmt-conf.sh ---> Running in 25444a779a58 Removing intermediate container 25444a779a58 ---> d22f87248659 Step 6/6 : ENTRYPOINT ["/register"] ---> Running in 32872dd66444 Removing intermediate container 32872dd66444 ---> be079407a8a9 Successfully built be079407a8a9 Successfully tagged multiarch/qemu-user-static:register
Below is the creatd images. The 1st line is register
tag's image.
$ docker image ls -a REPOSITORY TAG IMAGE ID CREATED SIZE multiarch/qemu-user-static register be079407a8a9 58 seconds ago 1.23MB <none> <none> d22f87248659 59 seconds ago 1.23MB <none> <none> 88c740e9bbe6 About a minute ago 1.21MB <none> <none> cc357a1c64ec About a minute ago 1.2MB <none> <none> 9861f53c0ee0 About a minute ago 1.2MB multiarch/qemu-user-static sparc64 f57ba065081f About a minute ago 1.31MB multiarch/qemu-user-static x86_64-sparc64 f57ba065081f About a minute ago 1.31MB multiarch/qemu-user-static sparc32plus fde067bc03dc About a minute ago 1.31MB multiarch/qemu-user-static x86_64-sparc32plus fde067bc03dc About a minute ago 1.31MB multiarch/qemu-user-static sparc e074528ec6d1 About a minute ago 1.28MB multiarch/qemu-user-static x86_64-sparc e074528ec6d1 About a minute ago 1.28MB multiarch/qemu-user-static sh4eb 81703749fd3e About a minute ago 1.26MB multiarch/qemu-user-static x86_64-sh4eb 81703749fd3e About a minute ago 1.26MB multiarch/qemu-user-static sh4 66881cd7937e About a minute ago 1.26MB multiarch/qemu-user-static x86_64-sh4 66881cd7937e About a minute ago 1.26MB multiarch/qemu-user-static s390x 1898e07e0156 About a minute ago 1.33MB multiarch/qemu-user-static x86_64-s390x 1898e07e0156 About a minute ago 1.33MB multiarch/qemu-user-static ppc64le c3823a9099f8 About a minute ago 1.58MB multiarch/qemu-user-static x86_64-ppc64le c3823a9099f8 About a minute ago 1.58MB multiarch/qemu-user-static ppc64abi32 41a04c8f61a9 About a minute ago 1.58MB multiarch/qemu-user-static x86_64-ppc64abi32 41a04c8f61a9 About a minute ago 1.58MB multiarch/qemu-user-static ppc64 3ee6a9a7322e About a minute ago 1.57MB multiarch/qemu-user-static x86_64-ppc64 3ee6a9a7322e About a minute ago 1.57MB multiarch/qemu-user-static ppc bc5b2536b632 About a minute ago 1.56MB multiarch/qemu-user-static x86_64-ppc bc5b2536b632 About a minute ago 1.56MB multiarch/qemu-user-static or1k 490ae085bfd2 2 minutes ago 1.24MB multiarch/qemu-user-static x86_64-or1k 490ae085bfd2 2 minutes ago 1.24MB multiarch/qemu-user-static nios2 163d3382c615 2 minutes ago 1.24MB multiarch/qemu-user-static x86_64-nios2 163d3382c615 2 minutes ago 1.24MB multiarch/qemu-user-static mipsn32el cd52dc63103e 2 minutes ago 1.45MB multiarch/qemu-user-static x86_64-mipsn32el cd52dc63103e 2 minutes ago 1.45MB multiarch/qemu-user-static mipsn32 75beaed6101d 2 minutes ago 1.46MB multiarch/qemu-user-static x86_64-mipsn32 75beaed6101d 2 minutes ago 1.46MB multiarch/qemu-user-static mipsel 3945f7b45a9a 2 minutes ago 1.43MB multiarch/qemu-user-static x86_64-mipsel 3945f7b45a9a 2 minutes ago 1.43MB multiarch/qemu-user-static mips64el aa170c25943f 2 minutes ago 1.45MB multiarch/qemu-user-static x86_64-mips64el aa170c25943f 2 minutes ago 1.45MB multiarch/qemu-user-static mips64 45482a0aca0f 2 minutes ago 1.45MB multiarch/qemu-user-static x86_64-mips64 45482a0aca0f 2 minutes ago 1.45MB multiarch/qemu-user-static mips a59b3106ea87 2 minutes ago 1.43MB multiarch/qemu-user-static x86_64-mips a59b3106ea87 2 minutes ago 1.43MB multiarch/qemu-user-static microblazeel b42dc2b63e2d 2 minutes ago 1.26MB multiarch/qemu-user-static x86_64-microblazeel b42dc2b63e2d 2 minutes ago 1.26MB multiarch/qemu-user-static microblaze 70a9418a4069 2 minutes ago 1.26MB multiarch/qemu-user-static x86_64-microblaze 70a9418a4069 2 minutes ago 1.26MB multiarch/qemu-user-static m68k 98065a9eea54 2 minutes ago 1.34MB multiarch/qemu-user-static x86_64-m68k 98065a9eea54 2 minutes ago 1.34MB multiarch/qemu-user-static i386 2587bc7b0b86 2 minutes ago 1.65MB multiarch/qemu-user-static x86_64-i386 2587bc7b0b86 2 minutes ago 1.65MB multiarch/qemu-user-static hppa ccb03e044476 3 minutes ago 1.27MB multiarch/qemu-user-static x86_64-hppa ccb03e044476 3 minutes ago 1.27MB multiarch/qemu-user-static cris 2ae51540cb3c 3 minutes ago 1.27MB multiarch/qemu-user-static x86_64-cris 2ae51540cb3c 3 minutes ago 1.27MB multiarch/qemu-user-static armeb 87e1e3f5d99b 3 minutes ago 1.46MB multiarch/qemu-user-static x86_64-armeb 87e1e3f5d99b 3 minutes ago 1.46MB multiarch/qemu-user-static arm 1a362b4aea82 3 minutes ago 1.45MB multiarch/qemu-user-static x86_64-arm 1a362b4aea82 3 minutes ago 1.45MB multiarch/qemu-user-static alpha 1f0b8b105e86 3 minutes ago 1.26MB multiarch/qemu-user-static x86_64-alpha 1f0b8b105e86 3 minutes ago 1.26MB multiarch/qemu-user-static aarch64 58b61c68fb02 3 minutes ago 1.61MB multiarch/qemu-user-static x86_64-aarch64 58b61c68fb02 3 minutes ago 1.61MB busybox latest af2f74c517aa 4 days ago 1.2MB
docker push
DO NOT RUN the commanddocker push multiarch/qemu-user-static
by yourself.
docker push multiarch/qemu-user-static
uploads the all the image to the DockerHub repository [4]
On next blog, I will dig the compatible images [5] such as ubuntu-core
and multiarch/ubuntu-debootstrap
and etc.
References
- [1] qemu-user-static: https://github.com/multiarch/qemu-user-static
- [2] qemu: http://www.qemu.org/
- [3] ci-multi-arch-test: https://github.com/junaruga/ci-multi-arch-test
- [4] qemu-user-static DockerHub: https://hub.docker.com/r/multiarch/qemu-user-static/tags
- [5] https://github.com/multiarch/qemu-user-static#compatible-images