Jun's Blog

Output, activities, memo and etc.

qemu-user-static - part 2 register

qemu-user-static enables us to run multi architecture environment such as ARM on x86_64 like this.

$ docker run --rm --privileged multiarch/qemu-user-static:register --reset

$ docker run --rm -t multiarch/ubuntu-debootstrap:arm64-bionic uname -a
Linux 28c784e9c7bc 4.4.0-101-generic #124~14.04.1-Ubuntu SMP Fri Nov 10 19:05:36 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux

On the part 1, we leaned the qemu-user-static is to create the small container including QEMU binary (qemu-foo-static) or the tar.gz file then upload it to the DockerHub container registry.

The part 2, we dig how the register container works reading the source code.

What does the register work?

First, register/Dockerfile file on [1] is the top level file. busybox is small Linux distribution.

FROM busybox
ENV QEMU_BIN_DIR=/usr/bin
ADD ./register.sh /register
ADD https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh
RUN chmod +x /qemu-binfmt-conf.sh
ENTRYPOINT ["/register"]

What is https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh?

Download it to check it.

$ wget https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh

It is a script managed at QEMU GitHub repository [2] https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh

Next ENTRYPOINT ["/register"] is called as the entry point. In above case, register.sh --reset is called.

register/register.sh

The steps in the file are

  1. Check if /proc/sys/fs/binfmt_misc directory exists
  2. mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc if /proc/sys/fs/binfmt_misc/register files does not exist
  3. find /proc/sys/fs/binfmt_misc -type f -name 'qemu-*' -exec sh -c 'echo -1 > {}' \; if --reset option is specified.
  4. exec /qemu-binfmt-conf.sh --qemu-suffix "-static" --qemu-path="${QEMU_BIN_DIR}" $@

Check if /proc/sys/fs/binfmt_misc directory exists

When I checked /proc/sys/fs/binfmt_misc/ directory on my host os (Fedora), below is the result.

$ ls -l /proc/sys/fs/binfmt_misc/
total 0
-rw-r--r-- 1 root root 0 Apr  7 14:55 qemu-aarch64
-rw-r--r-- 1 root root 0 Apr  7 14:55 qemu-aarch64_be
...
-rw-r--r-- 1 root root 0 Apr  7 14:55 qemu-xtensaeb
--w------- 1 root root 0 Apr  7 14:55 register
-rw-r--r-- 1 root root 0 Apr  7 14:55 status

So, step 2. mount binfmt_misc and 3. find .. is to create zero byte files /proc/sys/fs/binfmt_misc/qemu-*.

What is "binfmt_misc"? I found an article [3] [4]

According to [3]

The binfmt-support package contains a helper script to easily register/unregister binary formats with the kernel using the binfmt_misc module.

Install qemu, binfmt-support, and qemu-user-static:

apt-get install qemu binfmt-support qemu-user-static

There is a sub deb package of qemu deb package. https://packages.debian.org/sid/qemu-user-static

According to [4]

5.3.9.2. /proc/sys/fs/ This directory contains an array of options and information concerning various aspects of the file system, including quota, file handle, inode, and dentry information. The binfmt_misc/ directory is used to provide kernel support for miscellaneous binary formats.

exec /qemu-binfmt-conf.sh --qemu-suffix "-static" --qemu-path="${QEMU_BIN_DIR}" $@

Below is the actual command to be executed.

./qemu-binfmt-conf.sh --qemu-suffix "-static" --qemu-path="/usr/bin" --reset

Build the register container by myself to check the qemu-binfmt-conf.sh's behavior.

qemu-binfmt-conf.sh was downloaded above.

Add debug set -x.

$ vi register/qemu-binfmt-conf.sh
...
set -x
...

$ vi register/register.sh

$ git diff register/register.sh
 
set -x

QEMU_BIN_DIR=${QEMU_BIN_DIR:-/usr/bin}

$ vi register/Dockerfile
...
# ADD https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh <= Comment out this line.
ADD ./qemu-binfmt-conf.sh /qemu-binfmt-conf.sh <= Add this line.
RUN chmod +x /qemu-binfmt-conf.sh
ENTRYPOINT ["/register"]

$ docker build --rm -t junaruga/qemu-user-static:register register
...
Successfully tagged junaruga/qemu-user-static:register

$ docker image ls -a  | grep junaruga/qemu-user-static
junaruga/qemu-user-static    register              50384f9a6262        9 seconds ago       1.23MB

$ docker run --rm --privileged  junaruga/qemu-user-static:register --reset

Here is the output log.

Seeing qemu-binfmt-conf.sh buttom lines, the 2 main steps (functions) are below.

qemu_check_bintfmt_misc
qemu_set_binfmts

qemu_check_bintfmt_misc is to check valid system system status.

qemu_check_bintfmt_misc() {
    # load the binfmt_misc module
    if [ ! -d /proc/sys/fs/binfmt_misc ]; then
      if ! /sbin/modprobe binfmt_misc ; then
          exit 1
      fi
    fi
    if [ ! -f /proc/sys/fs/binfmt_misc/register ]; then
      if ! mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc ; then
          exit 1
      fi
    fi

    qemu_check_access /proc/sys/fs/binfmt_misc/register
}

qemu_target_list is for ${qemu_target_list}, run $BINFMT_SET (= qemu_register_interpreter function).

qemu_target_list="i386 i486 alpha arm armeb sparc32plus ppc ppc64 ppc64le m68k \
mips mipsel mipsn32 mipsn32el mips64 mips64el \
sh4 sh4eb s390x aarch64 aarch64_be hppa riscv32 riscv64 xtensa xtensaeb \
microblaze microblazeel or1k x86_64"
qemu_set_binfmts() {
    # probe cpu type
    host_family=$(qemu_get_family)

    # register the interpreter for each cpu except for the native one

    for cpu in ${qemu_target_list} ; do
        magic=$(eval echo \$${cpu}_magic)
        mask=$(eval echo \$${cpu}_mask)
        family=$(eval echo \$${cpu}_family)

        if [ "$magic" = "" ] || [ "$mask" = "" ] || [ "$family" = "" ] ; then
            echo "INTERNAL ERROR: unknown cpu $cpu" 1>&2
            continue
        fi

        qemu="$QEMU_PATH/qemu-$cpu"
        if [ "$cpu" = "i486" ] ; then
            qemu="$QEMU_PATH/qemu-i386"
        fi

        qemu="$qemu$QEMU_SUFFIX"
        if [ "$host_family" != "$family" ] ; then
            $BINFMT_SET
        fi
    done
}

qemu_register_interpreter registers a content to /proc/sys/fs/binfmt_misc/register procfs file. When a stdout content redirected to the file /proc/sys/fs/binfmt_misc/register. The bahabior of > is not like normal file.

qemu_register_interpreter() {
    echo "Setting $qemu as binfmt interpreter for $cpu"
    qemu_generate_register > /proc/sys/fs/binfmt_misc/register
}
qemu_generate_register() {
    flags=""
    if [ "$CREDENTIAL" = "yes" ] ; then
        flags="OC"
    fi
    if [ "$PERSISTENT" = "yes" ] ; then
        flags="${flags}F"
    fi

    echo ":qemu-$cpu:M::$magic:$mask:$qemu:$flags"
}

https://en.wikipedia.org/wiki/Binfmt_misc

The executable formats are registered through the special purpose file system binfmt_misc file-system interface (usually mounted under part of /proc). This is either done directly by sending special sequences to the register procfs file or using a wrapper like Debian-based distributions binfmt-support package[3] or systemd's systemd-binfmt.service[4][5].

Below code for each target cpu is what the register is doing.

echo ":qemu-$cpu:M::$magic:$mask:$qemu:$flags" > /proc/sys/fs/binfmt_misc/register

On next blog, I dig the compatible images. [5]

References

qemu-user-static - part 1 summary

qemu-user-static [1] is a software to enable qemu [2] consumable to build and run multi architecture (ARM 64-bit, ARM 32-bit, Intel 32-bit and etc) on x86_64 Intel processor environment.

If you run docker or a docker compatible software, you can run commands like this.

$ docker run --rm --privileged multiarch/qemu-user-static:register --reset

$ docker run --rm -t multiarch/ubuntu-debootstrap:arm64-bionic uname -a
Linux 28c784e9c7bc 4.4.0-101-generic #124~14.04.1-Ubuntu SMP Fri Nov 10 19:05:36 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux
$ docker run --rm --privileged multiarch/qemu-user-static:register --reset

$ docker run --rm -it multiarch/ubuntu-debootstrap:arm64-bionic bash

root@1e773d5d74af:/# uname -a
Linux 1e773d5d74af 5.0.5-200.fc29.x86_64 #1 SMP Wed Mar 27 20:58:04 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

root@1e773d5d74af:/# exit
exit

Why is this software important now?

Because of high growth of IoT that is to use a kind of mobile embedded device. HPC (super comuputing) on not only Intel CPU (x86_64), but also on ARM 64-bit CPU (aarch64), ARM 32-bit, IBM ppc64le, and etc. Open source project could be bulit on the multi-architecture environments.

But everyone does not have a environment for ARM and IBM architecture. Not only that, every CI (Continuous Integration) Service that is important for open source development does not support multi architecture. qemu-user-static enables users to run their software on the environments on local or CI "easily".

My motivaion to write this blog is that I want to spread this software to many people to promte softwares supporting the multi-architecture environments.

How does it work? (Reading the source code)

I did read the souce code to understand what it is doing.

Current latest version's commit hash is 20674ec (version v3.1.0-3 + 2 commits)

https://github.com/multiarch/qemu-user-static/commit/20674ec

$ git clone https://github.com/multiarch/qemu-user-static.git
$ cd qemu-user-static
$ git chckout 20674ec <= if you like reproducing below steps exactlly.

See .travis.yml. It is a top level file to understand it.

At first, install necessary packages.

If you are using Debian base Linux like Ubuntu, like this.

$ sudo apt-get install jq rpm2cpio cpio

If you are using RPM base Linux such as Fedora and CentOS, like this.

$ sudo dnf install jq cpio

or

$ sudo yum install jq cpio

Then download qemu RPM (pre-built binaries) file from Fedora Project. You can see all the built files from here. Right now f30 (Fedora 30) is the latest stable (beta) version.

$ wget --content-disposition https://kojipkgs.fedoraproject.org/packages/qemu/3.1.0/6.fc30/x86_64/qemu-user-static-3.1.0-6.fc30.x86_64.rpm

Extract downloaded RPM file.

$ rpm2cpio qemu-user-static-3.1.0-6.fc30.x86_64.rpm | cpio -dimv
./usr/bin/qemu-aarch64-static
./usr/bin/qemu-aarch64_be-static
./usr/bin/qemu-alpha-static
...

See extracted files under usr.

$ ls usr/
bin/  lib/  share/

$ ls usr/bin/
qemu-aarch64_be-static    qemu-mips64-static      qemu-riscv64-static
...

Then remaining steps are as following.

  • generate_tarballs.sh
  • publish.sh
  • update.sh
  • docker push

generate_tarballs.sh

generate_tarballs.sh copies ./usr/bin/qemu-*-static in the above extracted files, and creates tar.gz files to releases directory. Run below command with a shell debug mode sh -x to see each commands in the script file.

$ sh -x ./generate_tarballs.sh
+ rm -rf releases
+ mkdir -p releases
+ cp ./usr/bin/qemu-aarch64_be-static ./usr/bin/qemu-aarch64-static ... releases/
+ cd releases/
+ for file in *
+ tar -czf qemu-aarch64_be-static.tar.gz qemu-aarch64_be-static
+ cp qemu-aarch64_be-static.tar.gz x86_64_qemu-aarch64_be-static.tar.gz
...
+ tar -czf qemu-xtensa-static.tar.gz qemu-xtensa-static
+ cp qemu-xtensa-static.tar.gz x86_64_qemu-xtensa-static.tar.gz

See below files created under releases directory.

qemu-foo-static.tar.gz and x86_64_qemu-foo-static.tar.gz is same content right now.

$ ls releases | grep arch64
qemu-aarch64_be-static
qemu-aarch64_be-static.tar.gz
qemu-aarch64-static
qemu-aarch64-static.tar.gz
x86_64_qemu-aarch64_be-static.tar.gz
x86_64_qemu-aarch64-static.tar.gz

publish.sh

DO NOT RUN publish.sh by yourself. This script uploads all the file under the releases directory to GitHub release page like https://github.com/multiarch/qemu-user-static/releases/tag/v3.1.0-3 .

Check the uploaded command version. It is QEMU 3.1.

$ wget https://github.com/multiarch/qemu-user-static/releases/download/v3.1.0-3/qemu-aarch64-static

$ chmod +x qemu-aarch64-static 

$ ./qemu-aarch64-static --version
qemu-aarch64 version 3.1.0 (qemu-3.1.0-6.fc30)
Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers

update.sh

You can run update.sh on your local. This script creates container images. The each container images has /usr/bin/x86_64_qemu-foo-static.tar.gz

A content built by docker build is small. Just adding the command on the scratch (no content) container.

FROM scratch
ADD https://github.com/${REPO}/releases/download/v${VERSION}/${from_arch}_qemu-${to_arch}-static.tar.gz /usr/bin

Also the script creates a container by register/Dockerfile. busybox is a very small sized Linux distribution often used a container.

register/Dockerfile is like this. About what the file is doing, I will dig it on next blog.

FROM busybox
ENV QEMU_BIN_DIR=/usr/bin
ADD ./register.sh /register
ADD https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh
RUN chmod +x /qemu-binfmt-conf.sh
ENTRYPOINT ["/register"]
$ docker build -t ${REPO}:register register

So, actually build it on local.

$ sh -x ./update.sh -v "3.1.0-3" -r "multiarch/qemu-user-static"
...
+ from_arch=x86_64
+ to_archs='aarch64 alpha arm armeb cris hppa i386 m68k microblaze microblazeel mips mips64 mips64el mipsel mipsn32 mipsn32el nios2 or1k ppc ppc64 ppc64abi32 ppc64le s390x sh4 sh4eb sparc sparc32plus sparc64 x86_64'
...
+ for to_arch in $to_archs
+ '[' x86_64 '!=' aarch64 ']'
+ docker build -t multiarch/qemu-user-static:x86_64-aarch64 -
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM scratch
 ---> 
Step 2/2 : ADD https://github.com/multiarch/qemu-user-static/releases/download/v3.1.0-3/x86_64_qemu-aarch64-static.tar.gz /usr/bin
Downloading   1.61MB/1.61MB
 ---> 58b61c68fb02
Successfully built 58b61c68fb02
Successfully tagged multiarch/qemu-user-static:x86_64-aarch64
+ docker tag multiarch/qemu-user-static:x86_64-aarch64 multiarch/qemu-user-static:aarch64
...
+ docker build -t multiarch/qemu-user-static:register register
Sending build context to Docker daemon  3.584kB
Step 1/6 : FROM busybox
latest: Pulling from library/busybox
fc1a6b909f82: Pull complete 
Digest: sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd
Status: Downloaded newer image for busybox:latest
 ---> af2f74c517aa
Step 2/6 : ENV QEMU_BIN_DIR=/usr/bin
 ---> Running in 19568e18a181
Removing intermediate container 19568e18a181
 ---> 9861f53c0ee0
Step 3/6 : ADD ./register.sh /register
 ---> cc357a1c64ec
Step 4/6 : ADD https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh
Downloading  13.24kB
 ---> 88c740e9bbe6
Step 5/6 : RUN chmod +x /qemu-binfmt-conf.sh
 ---> Running in 25444a779a58
Removing intermediate container 25444a779a58
 ---> d22f87248659
Step 6/6 : ENTRYPOINT ["/register"]
 ---> Running in 32872dd66444
Removing intermediate container 32872dd66444
 ---> be079407a8a9
Successfully built be079407a8a9
Successfully tagged multiarch/qemu-user-static:register

Below is the creatd images. The 1st line is register tag's image.

$ docker image ls -a
REPOSITORY                   TAG                   IMAGE ID            CREATED              SIZE
multiarch/qemu-user-static   register              be079407a8a9        58 seconds ago       1.23MB
<none>                       <none>                d22f87248659        59 seconds ago       1.23MB
<none>                       <none>                88c740e9bbe6        About a minute ago   1.21MB
<none>                       <none>                cc357a1c64ec        About a minute ago   1.2MB
<none>                       <none>                9861f53c0ee0        About a minute ago   1.2MB
multiarch/qemu-user-static   sparc64               f57ba065081f        About a minute ago   1.31MB
multiarch/qemu-user-static   x86_64-sparc64        f57ba065081f        About a minute ago   1.31MB
multiarch/qemu-user-static   sparc32plus           fde067bc03dc        About a minute ago   1.31MB
multiarch/qemu-user-static   x86_64-sparc32plus    fde067bc03dc        About a minute ago   1.31MB
multiarch/qemu-user-static   sparc                 e074528ec6d1        About a minute ago   1.28MB
multiarch/qemu-user-static   x86_64-sparc          e074528ec6d1        About a minute ago   1.28MB
multiarch/qemu-user-static   sh4eb                 81703749fd3e        About a minute ago   1.26MB
multiarch/qemu-user-static   x86_64-sh4eb          81703749fd3e        About a minute ago   1.26MB
multiarch/qemu-user-static   sh4                   66881cd7937e        About a minute ago   1.26MB
multiarch/qemu-user-static   x86_64-sh4            66881cd7937e        About a minute ago   1.26MB
multiarch/qemu-user-static   s390x                 1898e07e0156        About a minute ago   1.33MB
multiarch/qemu-user-static   x86_64-s390x          1898e07e0156        About a minute ago   1.33MB
multiarch/qemu-user-static   ppc64le               c3823a9099f8        About a minute ago   1.58MB
multiarch/qemu-user-static   x86_64-ppc64le        c3823a9099f8        About a minute ago   1.58MB
multiarch/qemu-user-static   ppc64abi32            41a04c8f61a9        About a minute ago   1.58MB
multiarch/qemu-user-static   x86_64-ppc64abi32     41a04c8f61a9        About a minute ago   1.58MB
multiarch/qemu-user-static   ppc64                 3ee6a9a7322e        About a minute ago   1.57MB
multiarch/qemu-user-static   x86_64-ppc64          3ee6a9a7322e        About a minute ago   1.57MB
multiarch/qemu-user-static   ppc                   bc5b2536b632        About a minute ago   1.56MB
multiarch/qemu-user-static   x86_64-ppc            bc5b2536b632        About a minute ago   1.56MB
multiarch/qemu-user-static   or1k                  490ae085bfd2        2 minutes ago        1.24MB
multiarch/qemu-user-static   x86_64-or1k           490ae085bfd2        2 minutes ago        1.24MB
multiarch/qemu-user-static   nios2                 163d3382c615        2 minutes ago        1.24MB
multiarch/qemu-user-static   x86_64-nios2          163d3382c615        2 minutes ago        1.24MB
multiarch/qemu-user-static   mipsn32el             cd52dc63103e        2 minutes ago        1.45MB
multiarch/qemu-user-static   x86_64-mipsn32el      cd52dc63103e        2 minutes ago        1.45MB
multiarch/qemu-user-static   mipsn32               75beaed6101d        2 minutes ago        1.46MB
multiarch/qemu-user-static   x86_64-mipsn32        75beaed6101d        2 minutes ago        1.46MB
multiarch/qemu-user-static   mipsel                3945f7b45a9a        2 minutes ago        1.43MB
multiarch/qemu-user-static   x86_64-mipsel         3945f7b45a9a        2 minutes ago        1.43MB
multiarch/qemu-user-static   mips64el              aa170c25943f        2 minutes ago        1.45MB
multiarch/qemu-user-static   x86_64-mips64el       aa170c25943f        2 minutes ago        1.45MB
multiarch/qemu-user-static   mips64                45482a0aca0f        2 minutes ago        1.45MB
multiarch/qemu-user-static   x86_64-mips64         45482a0aca0f        2 minutes ago        1.45MB
multiarch/qemu-user-static   mips                  a59b3106ea87        2 minutes ago        1.43MB
multiarch/qemu-user-static   x86_64-mips           a59b3106ea87        2 minutes ago        1.43MB
multiarch/qemu-user-static   microblazeel          b42dc2b63e2d        2 minutes ago        1.26MB
multiarch/qemu-user-static   x86_64-microblazeel   b42dc2b63e2d        2 minutes ago        1.26MB
multiarch/qemu-user-static   microblaze            70a9418a4069        2 minutes ago        1.26MB
multiarch/qemu-user-static   x86_64-microblaze     70a9418a4069        2 minutes ago        1.26MB
multiarch/qemu-user-static   m68k                  98065a9eea54        2 minutes ago        1.34MB
multiarch/qemu-user-static   x86_64-m68k           98065a9eea54        2 minutes ago        1.34MB
multiarch/qemu-user-static   i386                  2587bc7b0b86        2 minutes ago        1.65MB
multiarch/qemu-user-static   x86_64-i386           2587bc7b0b86        2 minutes ago        1.65MB
multiarch/qemu-user-static   hppa                  ccb03e044476        3 minutes ago        1.27MB
multiarch/qemu-user-static   x86_64-hppa           ccb03e044476        3 minutes ago        1.27MB
multiarch/qemu-user-static   cris                  2ae51540cb3c        3 minutes ago        1.27MB
multiarch/qemu-user-static   x86_64-cris           2ae51540cb3c        3 minutes ago        1.27MB
multiarch/qemu-user-static   armeb                 87e1e3f5d99b        3 minutes ago        1.46MB
multiarch/qemu-user-static   x86_64-armeb          87e1e3f5d99b        3 minutes ago        1.46MB
multiarch/qemu-user-static   arm                   1a362b4aea82        3 minutes ago        1.45MB
multiarch/qemu-user-static   x86_64-arm            1a362b4aea82        3 minutes ago        1.45MB
multiarch/qemu-user-static   alpha                 1f0b8b105e86        3 minutes ago        1.26MB
multiarch/qemu-user-static   x86_64-alpha          1f0b8b105e86        3 minutes ago        1.26MB
multiarch/qemu-user-static   aarch64               58b61c68fb02        3 minutes ago        1.61MB
multiarch/qemu-user-static   x86_64-aarch64        58b61c68fb02        3 minutes ago        1.61MB
busybox                      latest                af2f74c517aa        4 days ago           1.2MB

docker push

DO NOT RUN the commanddocker push multiarch/qemu-user-static by yourself.

docker push multiarch/qemu-user-static uploads the all the image to the DockerHub repository [4]

On next blog, I will dig the compatible images [5] such as ubuntu-core and multiarch/ubuntu-debootstrap and etc.

References