Another Japan in the World

Jun Aruga's blog.

qemu-user-static - part 1 summary

qemu-user-static [1] is a software to enable qemu [2] consumable to build and run multi architecture (ARM 64-bit, ARM 32-bit, Intel 32-bit and etc) on x86_64 Intel processor environment.

If you run docker or a docker compatible software, you can run commands like this.

$ docker run --rm --privileged multiarch/qemu-user-static:register --reset

$ docker run --rm -t multiarch/ubuntu-debootstrap:arm64-bionic uname -a
Linux 28c784e9c7bc 4.4.0-101-generic #124~14.04.1-Ubuntu SMP Fri Nov 10 19:05:36 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux
$ docker run --rm --privileged multiarch/qemu-user-static:register --reset

$ docker run --rm -it multiarch/ubuntu-debootstrap:arm64-bionic bash

root@1e773d5d74af:/# uname -a
Linux 1e773d5d74af 5.0.5-200.fc29.x86_64 #1 SMP Wed Mar 27 20:58:04 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

root@1e773d5d74af:/# exit
exit

Why is this software important now?

Because of high growth of IoT that is to use a kind of mobile embedded device. HPC (super comuputing) on not only Intel CPU (x86_64), but also on ARM 64-bit CPU (aarch64), ARM 32-bit, IBM ppc64le, and etc. Open source project could be bulit on the multi-architecture environments.

But everyone does not have a environment for ARM and IBM architecture. Not only that, every CI (Continuous Integration) Service that is important for open source development does not support multi architecture. qemu-user-static enables users to run their software on the environments on local or CI "easily".

My motivaion to write this blog is that I want to spread this software to many people to promte softwares supporting the multi-architecture environments.

How does it work? (Reading the source code)

I did read the souce code to understand what it is doing.

Current latest version's commit hash is 20674ec (version v3.1.0-3 + 2 commits)

https://github.com/multiarch/qemu-user-static/commit/20674ec

$ git clone https://github.com/multiarch/qemu-user-static.git
$ cd qemu-user-static
$ git chckout 20674ec <= if you like reproducing below steps exactlly.

See .travis.yml. It is a top level file to understand it.

At first, install necessary packages.

If you are using Debian base Linux like Ubuntu, like this.

$ sudo apt-get install jq rpm2cpio cpio

If you are using RPM base Linux such as Fedora and CentOS, like this.

$ sudo dnf install jq cpio

or

$ sudo yum install jq cpio

Then download qemu RPM (pre-built binaries) file from Fedora Project. You can see all the built files from here. Right now f30 (Fedora 30) is the latest stable (beta) version.

$ wget --content-disposition https://kojipkgs.fedoraproject.org/packages/qemu/3.1.0/6.fc30/x86_64/qemu-user-static-3.1.0-6.fc30.x86_64.rpm

Extract downloaded RPM file.

$ rpm2cpio qemu-user-static-3.1.0-6.fc30.x86_64.rpm | cpio -dimv
./usr/bin/qemu-aarch64-static
./usr/bin/qemu-aarch64_be-static
./usr/bin/qemu-alpha-static
...

See extracted files under usr.

$ ls usr/
bin/  lib/  share/

$ ls usr/bin/
qemu-aarch64_be-static    qemu-mips64-static      qemu-riscv64-static
...

Then remaining steps are as following.

  • generate_tarballs.sh
  • publish.sh
  • update.sh
  • docker push

generate_tarballs.sh

generate_tarballs.sh copies ./usr/bin/qemu-*-static in the above extracted files, and creates tar.gz files to releases directory. Run below command with a shell debug mode sh -x to see each commands in the script file.

$ sh -x ./generate_tarballs.sh
+ rm -rf releases
+ mkdir -p releases
+ cp ./usr/bin/qemu-aarch64_be-static ./usr/bin/qemu-aarch64-static ... releases/
+ cd releases/
+ for file in *
+ tar -czf qemu-aarch64_be-static.tar.gz qemu-aarch64_be-static
+ cp qemu-aarch64_be-static.tar.gz x86_64_qemu-aarch64_be-static.tar.gz
...
+ tar -czf qemu-xtensa-static.tar.gz qemu-xtensa-static
+ cp qemu-xtensa-static.tar.gz x86_64_qemu-xtensa-static.tar.gz

See below files created under releases directory.

qemu-foo-static.tar.gz and x86_64_qemu-foo-static.tar.gz is same content right now.

$ ls releases | grep arch64
qemu-aarch64_be-static
qemu-aarch64_be-static.tar.gz
qemu-aarch64-static
qemu-aarch64-static.tar.gz
x86_64_qemu-aarch64_be-static.tar.gz
x86_64_qemu-aarch64-static.tar.gz

publish.sh

DO NOT RUN publish.sh by yourself. This script uploads all the file under the releases directory to GitHub release page like https://github.com/multiarch/qemu-user-static/releases/tag/v3.1.0-3 .

Check the uploaded command version. It is QEMU 3.1.

$ wget https://github.com/multiarch/qemu-user-static/releases/download/v3.1.0-3/qemu-aarch64-static

$ chmod +x qemu-aarch64-static 

$ ./qemu-aarch64-static --version
qemu-aarch64 version 3.1.0 (qemu-3.1.0-6.fc30)
Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers

update.sh

You can run update.sh on your local. This script creates container images. The each container images has /usr/bin/x86_64_qemu-foo-static.tar.gz

A content built by docker build is small. Just adding the command on the scratch (no content) container.

FROM scratch
ADD https://github.com/${REPO}/releases/download/v${VERSION}/${from_arch}_qemu-${to_arch}-static.tar.gz /usr/bin

Also the script creates a container by register/Dockerfile. busybox is a very small sized Linux distribution often used a container.

register/Dockerfile is like this. About what the file is doing, I will dig it on next blog.

FROM busybox
ENV QEMU_BIN_DIR=/usr/bin
ADD ./register.sh /register
ADD https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh
RUN chmod +x /qemu-binfmt-conf.sh
ENTRYPOINT ["/register"]
$ docker build -t ${REPO}:register register

So, actually build it on local.

$ sh -x ./update.sh -v "3.1.0-3" -r "multiarch/qemu-user-static"
...
+ from_arch=x86_64
+ to_archs='aarch64 alpha arm armeb cris hppa i386 m68k microblaze microblazeel mips mips64 mips64el mipsel mipsn32 mipsn32el nios2 or1k ppc ppc64 ppc64abi32 ppc64le s390x sh4 sh4eb sparc sparc32plus sparc64 x86_64'
...
+ for to_arch in $to_archs
+ '[' x86_64 '!=' aarch64 ']'
+ docker build -t multiarch/qemu-user-static:x86_64-aarch64 -
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM scratch
 ---> 
Step 2/2 : ADD https://github.com/multiarch/qemu-user-static/releases/download/v3.1.0-3/x86_64_qemu-aarch64-static.tar.gz /usr/bin
Downloading   1.61MB/1.61MB
 ---> 58b61c68fb02
Successfully built 58b61c68fb02
Successfully tagged multiarch/qemu-user-static:x86_64-aarch64
+ docker tag multiarch/qemu-user-static:x86_64-aarch64 multiarch/qemu-user-static:aarch64
...
+ docker build -t multiarch/qemu-user-static:register register
Sending build context to Docker daemon  3.584kB
Step 1/6 : FROM busybox
latest: Pulling from library/busybox
fc1a6b909f82: Pull complete 
Digest: sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd
Status: Downloaded newer image for busybox:latest
 ---> af2f74c517aa
Step 2/6 : ENV QEMU_BIN_DIR=/usr/bin
 ---> Running in 19568e18a181
Removing intermediate container 19568e18a181
 ---> 9861f53c0ee0
Step 3/6 : ADD ./register.sh /register
 ---> cc357a1c64ec
Step 4/6 : ADD https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh /qemu-binfmt-conf.sh
Downloading  13.24kB
 ---> 88c740e9bbe6
Step 5/6 : RUN chmod +x /qemu-binfmt-conf.sh
 ---> Running in 25444a779a58
Removing intermediate container 25444a779a58
 ---> d22f87248659
Step 6/6 : ENTRYPOINT ["/register"]
 ---> Running in 32872dd66444
Removing intermediate container 32872dd66444
 ---> be079407a8a9
Successfully built be079407a8a9
Successfully tagged multiarch/qemu-user-static:register

Below is the creatd images. The 1st line is register tag's image.

$ docker image ls -a
REPOSITORY                   TAG                   IMAGE ID            CREATED              SIZE
multiarch/qemu-user-static   register              be079407a8a9        58 seconds ago       1.23MB
<none>                       <none>                d22f87248659        59 seconds ago       1.23MB
<none>                       <none>                88c740e9bbe6        About a minute ago   1.21MB
<none>                       <none>                cc357a1c64ec        About a minute ago   1.2MB
<none>                       <none>                9861f53c0ee0        About a minute ago   1.2MB
multiarch/qemu-user-static   sparc64               f57ba065081f        About a minute ago   1.31MB
multiarch/qemu-user-static   x86_64-sparc64        f57ba065081f        About a minute ago   1.31MB
multiarch/qemu-user-static   sparc32plus           fde067bc03dc        About a minute ago   1.31MB
multiarch/qemu-user-static   x86_64-sparc32plus    fde067bc03dc        About a minute ago   1.31MB
multiarch/qemu-user-static   sparc                 e074528ec6d1        About a minute ago   1.28MB
multiarch/qemu-user-static   x86_64-sparc          e074528ec6d1        About a minute ago   1.28MB
multiarch/qemu-user-static   sh4eb                 81703749fd3e        About a minute ago   1.26MB
multiarch/qemu-user-static   x86_64-sh4eb          81703749fd3e        About a minute ago   1.26MB
multiarch/qemu-user-static   sh4                   66881cd7937e        About a minute ago   1.26MB
multiarch/qemu-user-static   x86_64-sh4            66881cd7937e        About a minute ago   1.26MB
multiarch/qemu-user-static   s390x                 1898e07e0156        About a minute ago   1.33MB
multiarch/qemu-user-static   x86_64-s390x          1898e07e0156        About a minute ago   1.33MB
multiarch/qemu-user-static   ppc64le               c3823a9099f8        About a minute ago   1.58MB
multiarch/qemu-user-static   x86_64-ppc64le        c3823a9099f8        About a minute ago   1.58MB
multiarch/qemu-user-static   ppc64abi32            41a04c8f61a9        About a minute ago   1.58MB
multiarch/qemu-user-static   x86_64-ppc64abi32     41a04c8f61a9        About a minute ago   1.58MB
multiarch/qemu-user-static   ppc64                 3ee6a9a7322e        About a minute ago   1.57MB
multiarch/qemu-user-static   x86_64-ppc64          3ee6a9a7322e        About a minute ago   1.57MB
multiarch/qemu-user-static   ppc                   bc5b2536b632        About a minute ago   1.56MB
multiarch/qemu-user-static   x86_64-ppc            bc5b2536b632        About a minute ago   1.56MB
multiarch/qemu-user-static   or1k                  490ae085bfd2        2 minutes ago        1.24MB
multiarch/qemu-user-static   x86_64-or1k           490ae085bfd2        2 minutes ago        1.24MB
multiarch/qemu-user-static   nios2                 163d3382c615        2 minutes ago        1.24MB
multiarch/qemu-user-static   x86_64-nios2          163d3382c615        2 minutes ago        1.24MB
multiarch/qemu-user-static   mipsn32el             cd52dc63103e        2 minutes ago        1.45MB
multiarch/qemu-user-static   x86_64-mipsn32el      cd52dc63103e        2 minutes ago        1.45MB
multiarch/qemu-user-static   mipsn32               75beaed6101d        2 minutes ago        1.46MB
multiarch/qemu-user-static   x86_64-mipsn32        75beaed6101d        2 minutes ago        1.46MB
multiarch/qemu-user-static   mipsel                3945f7b45a9a        2 minutes ago        1.43MB
multiarch/qemu-user-static   x86_64-mipsel         3945f7b45a9a        2 minutes ago        1.43MB
multiarch/qemu-user-static   mips64el              aa170c25943f        2 minutes ago        1.45MB
multiarch/qemu-user-static   x86_64-mips64el       aa170c25943f        2 minutes ago        1.45MB
multiarch/qemu-user-static   mips64                45482a0aca0f        2 minutes ago        1.45MB
multiarch/qemu-user-static   x86_64-mips64         45482a0aca0f        2 minutes ago        1.45MB
multiarch/qemu-user-static   mips                  a59b3106ea87        2 minutes ago        1.43MB
multiarch/qemu-user-static   x86_64-mips           a59b3106ea87        2 minutes ago        1.43MB
multiarch/qemu-user-static   microblazeel          b42dc2b63e2d        2 minutes ago        1.26MB
multiarch/qemu-user-static   x86_64-microblazeel   b42dc2b63e2d        2 minutes ago        1.26MB
multiarch/qemu-user-static   microblaze            70a9418a4069        2 minutes ago        1.26MB
multiarch/qemu-user-static   x86_64-microblaze     70a9418a4069        2 minutes ago        1.26MB
multiarch/qemu-user-static   m68k                  98065a9eea54        2 minutes ago        1.34MB
multiarch/qemu-user-static   x86_64-m68k           98065a9eea54        2 minutes ago        1.34MB
multiarch/qemu-user-static   i386                  2587bc7b0b86        2 minutes ago        1.65MB
multiarch/qemu-user-static   x86_64-i386           2587bc7b0b86        2 minutes ago        1.65MB
multiarch/qemu-user-static   hppa                  ccb03e044476        3 minutes ago        1.27MB
multiarch/qemu-user-static   x86_64-hppa           ccb03e044476        3 minutes ago        1.27MB
multiarch/qemu-user-static   cris                  2ae51540cb3c        3 minutes ago        1.27MB
multiarch/qemu-user-static   x86_64-cris           2ae51540cb3c        3 minutes ago        1.27MB
multiarch/qemu-user-static   armeb                 87e1e3f5d99b        3 minutes ago        1.46MB
multiarch/qemu-user-static   x86_64-armeb          87e1e3f5d99b        3 minutes ago        1.46MB
multiarch/qemu-user-static   arm                   1a362b4aea82        3 minutes ago        1.45MB
multiarch/qemu-user-static   x86_64-arm            1a362b4aea82        3 minutes ago        1.45MB
multiarch/qemu-user-static   alpha                 1f0b8b105e86        3 minutes ago        1.26MB
multiarch/qemu-user-static   x86_64-alpha          1f0b8b105e86        3 minutes ago        1.26MB
multiarch/qemu-user-static   aarch64               58b61c68fb02        3 minutes ago        1.61MB
multiarch/qemu-user-static   x86_64-aarch64        58b61c68fb02        3 minutes ago        1.61MB
busybox                      latest                af2f74c517aa        4 days ago           1.2MB

docker push

DO NOT RUN the commanddocker push multiarch/qemu-user-static by yourself.

docker push multiarch/qemu-user-static uploads the all the image to the DockerHub repository [4]

On next blog, I will dig the compatible images [5] such as ubuntu-core and multiarch/ubuntu-debootstrap and etc.

References