Running cross-arch container images on your linux laptop

With the introduction of Docker for Mac, I ran across an exciting blog post: http://blog.hypriot.com/post/first-touch-down-with-docker-for-mac. I don't use a Mac for development, but what made that blog post interesting to me was the "Easter Egg" bit. It was titled, "There is another big ARM surprise," which is pretty sweet (so hopefully you've read that by now).

But what about other architectures? And what about not just doing this in a Mac? Well, get your ribboned baskets ready, because that Easter Egg has led me to a giant Easter Egg minefield of awesome. There are some folks over at Scaleway working on a multiarch project, and they've put together two key things:

1. All the Easter Eggs: [scroll to Downloads] https://github.com/multiarch/qemu-user-static/releases
2. The prep your Easter basket needs to use them: https://hub.docker.com/r/multiarch/qemu-user-static

So let's back up a little and talk about what is going on here. Some of this was mentioned in the blog post I referenced at the beginning of this post, but I think a bit more exploration is fun. (If you don't, skip down to the teal deer).

In Linux, there's a binary that allows you to run ELF binaries that weren't compiled for the architecture you are running on. It's called binfmt_misc, and you can read more about it here: https://www.kernel.org/doc/Documentation/binfmt_misc.txt.That binary doesn't actually run the program. It just provides the mechanism to make sure the right interpreter does, based on some bits embedded in the program itself.  The binfmt_misc binary checks the magic bits, then cross-references with what it finds in /proc/sys/fs/binfmt_misc/ for what to do.

That's where #1 comes in. binfmt_misc comes shipped with most Linux distros, but it can't do the job alone. It needs an interpreter. And what better interepreter than qemu? The multiarch project includes over a dozen compiled static qemu binaries! There's more than just the ARM qemu binary in there, so whatever architecture you want to run, I bet it's in their list.

But you can't just plop the emulator into your system and start running ARM or POWER containers. You've got to let binfmt_misc know what binaries should do what. You've got to set up those magic numbers, and also have them point to the right place. That's where #2 is fantastic. Not only are all the strings that binfmt_misc needs already assembled, the Scaleway folks created a docker container that will add them all to your host! If they hadn't you'd have to put together strings like  

:ppc64le:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\x00:'${QEMU_BIN_DIR}'/qemu-ppc64le-static:

and then get them in the right place in your fs. Instead, you just run:
$ docker run --rm --privileged multiarch/qemu-user-static:register

and you're set up! 

You can check that these were added by:
$ ls /proc/sys/fs/binfmt_misc/
aarch64  arm    kshcomp  mips    mips64el  mipsn32    ppc    ppc64le   s390x  sh4eb  status alpha    armeb  m68k     mips64  mipsel    mipsn32el  ppc64  register  sh4    sparc
 
(Note: You do also have to have binfmt_misc mounted on your system, but I'm leaving that step out because on my F23 workstation it was mounted by default.)

All that's left is running the container. But the container needs access to the emulator, so you can just bind-mount it at runtime (e.g. with docker's -v).

So now, for those of you who stayed with me, thanks. For everyone else, it's deer time.

Oh hai! Let's go:

• Download https://github.com/multiarch/qemu-user-static/releases/download/v2.8.1/x86_64_qemu-ppc64le-static.tar.gz -o x86_64_qemu-ppc64le-static.tar.gz
•  $ docker run --rm --privileged multiarch/qemu-user-static:register
•  $ docker run -it --rm -v /home/yournamehere/Downloads/qemu-ppc64le-static:/usr/bin/qemu-ppc64le-static ppc64le/busybox:latest uname -a

Tada!

My example was with ppc64le, but you can download one of the other qemu binaries in the first step, depending on intended arch of the container you want run.