Emulating aarch64

Finally building SBCL on aarch64 arm platforms! · June, 4, 2020

It's here! It's in the testing repo! We can all rejoice, for we have SBCL 2.0.5 on arm64 on Alpine! All of the musl libc glory, right in time for the latest batch of PinePhones. I'm excited to say the least thinking I could get to run my favorite lisp implementation on the PinePhone!

I might be one of maybe a dozen people who feels that way, but I'm elated nonetheless. Since I picked up the maintenance of SBCL for Alpine I've dreamed of getting it on aarch64. Being able to use it on my old Nokia N900 feels powerful, but a newer more modern hardware revision which is slowly become more and more prevalent in the large scale infrastructure scene has potential!

And I could not have done any of it without the hard work of Eric Timmons. Eric is the author of the musl libc patches that enabled me to first get SBCL building on Alpine was it was abandoned. He has been trying for over a year to get these adopted upstream for his own sake, and without them it would be impossible to run SBCL on Alpine. I am eternally grateful for his efforts, and his willingness to discuss and work towards getting these up-streamed. The FOSS community is an absolute delight to work in, and he is an exemplary example.

But we're not here to listen to me gush about how much I love FOSS (or are we?), we're here for gritty technical nonsense.

Technical

One of the hurdles I ran into early on was that I don't actually own that many non-x86_64 systems. And really, don't try to compile SBCL on an old Nokia. I tried, a few times, it is painful beyond words. Which means the only solution is to use QEMU, or I mean buy pine boards (which absolutely are on their way!).

With qemu-static binaries you can build docker containers that act as little isolated architecture specific builders, absolutely great for CI/CD. Unless you're running your CI/CD on a $5 Digital Ocean droplet like I am, then your mileage on anything complex is pretty limited. Qemu-system-aarch64 is your friend, and one that I owe Eric for suggesting.

qemu-system-aarch64 \
  -M virt -m 4096M -cpu cortex-a53 -smp 6\
  -kernel vmlinuz-lts -initrd initramfs-lts \
  -drive file=sbclaarch64.config,if=virtio \
  -append "console=ttyAMA0 ip=dhcp alpine_repo=http://dl-cdn.alpinelinux.org/alpine/edge/main/" \
  -nographic
 

With qemu-system you can do something crazy like pass the vmlinuz and initramfs alongside an Alpine repo and boot into an Alpine Linux installation session, without ever needing to pull down an ISO image in the first place. But that's just half the magic, if you pass it a img file, format it as vfat, and mount it inside of the VM, then you can leverage Alpine's LBU system to create a self building base VM. AKA the disposableness of a docker image, but with a lot more interactivity and control.

After running through the setup-alpine steps in the aarch64 VM, I went about persisting the basics. A few necessary packages (linux kernel, sudo, micro emacs, abuild toolchain), a user, and the few configurations needed. With lbu it's ridiculously simple to do something like generate an auild signing key, and persist it between in RAM systems. The apkvol essentially is tarball which gets overlayed on top of the tmpfs of the Alpine system running in RAM. When you provide that drive to the qemu-system vm, and provide it during setup-alpine, you're building a small offline package repository and data store to rebuild your from RAM systems. Superbly interesting technology.

For anyone unfamiliar getting that setup looks a little something like this:

• qemu-img create -f raw lbuvol.img 400
• qemu-system-aarch64 ...
• Inside the VM format /dev/vda with fdisk
• mkdofs /dev/vda1 && mkdir -P /media/vda1
• setup-alpine, passing vda1 as the storage volume

Really after that point going about your business setting things up as you need as normal is the way to go. Once you've got the in RAM VM the way you want it run a lbu commit to make the initial apkvol, and anything you want to add custom (like say a users .abuild, or .ssh) you can persist with lbu add /path/to/thing.

Compiling SBCL 2.0.5 in such a system took 6 hours, running on a host with an i7-6600, and providing the VM with a dual core cortex-a53, and 3Gb of RAM. The same compilation attempts in Docker on top of a Digital Ocean droplet took upwards of 3 days, often times simply stalling out and dying completely during compilation. Going forward I'll be making heavy use of qemu-system for manual builds, 6 hours seems ridiculously speedy in comparison!