((λ (x) (create x)) '(knowledge))

Building custom incus images

Just so I can adopt Salt extensions · June 27th, 2025

Right, so in the wake of Broadcom's purchase of VMWare was the additional acquisition of the SaltStack project. This isn't surprising to anyone using Salt, since it was well communicated that VMWare had acquired Salt previously. But that little bit of information got somewhat drown out since Broadcom's focus has been on VMWare's hypervisor software, and that entire ordeal has caused quite a lot of noise. Sorry for anyone having to deal with it, maybe you should move some of your workloads to Incus? No it isn't comparable, but it's a better tool.

That's not what we're hear to talk about though, well the Incus part yes, Broadcom no; we've all heard enough of that already. But because Broadcom acquired SaltStack, and then gutted the development team, which forced them into massively purging modules from the Salt code base in the name of future maintainability, there is this void in the ecosystem. A lot like what happened with Ansible, and Puppet, we now have a slew of modules (~750 total) that will be either abandoned or adopted and maintained by the community. Which as you can assume by this post affects me personally.

See I maintain SaltStack for Alpine, and while that hasn't been a smooth and problem free process, it's one I'm proud of and leverage a ton. I would really like to continue to use the modules I depend on, like the APK package handling & S3 file storage backend. I also want to add my own extensions, because that's like the entire selling point of SaltStack. It isn't just a state execution system, I could pick up an RMM if that's what I wanted. Rather Salt is this neat combination of remote execution, built in such a way as to be entirely modular and extensible. Everything is an extension or plugin built on top of a module which gets lazy loaded at execution time. Which means I can write custom discrete scenario specific logic into the system itself to address truly bizarre scenarios. That level of flexibility however comes with a price, and that price is technical complexity. For my desires the price I pay is having to learn how to.

Fortunately, that's right up my alley.

Despite that fact, I've been warily eyeing the project since the acquisition, hoping someone would adopt the packages I need so I wouldn't need to do anything. Always nice to benefit from someone else's hard work right? But after a yearish of waiting, it's pretty clear nobody is going to step up to the plate and do what I need. So I need to! I guess I'm well positioned to do it, I'm appropriately incentivized as a consumer of the software & I have the resources needed to materially affect the distribution I maintain Salt for. No brainer, I'll at least make sure my needs are met. The price I pay is more open source development, entirely worth it.

Just, one itty bitty tiny problem. The migration tooling, and complex testing & git commit workflows depend on python 3.10 which Alpine stopped shipping forever ago..

And that my friends is where this story ended about 6ish months ago. Barrier to entry was too high, so I gave up. I really don't like Python so I was demotivated, and it was still unclear whether or not the project would have life breathed back into it. Yet here we are, we've gotten 4 releases of salt on both the LTS and STS branches in the last month-ish, and things are feeling hopeful! Hopefully enough that I decided to take another crack at this.

Building Images with Incus

So my hypervisor of choice is Incus, it's a delightful little cli first tool for container and VM orchestration. System containers specifically versus the OCI style ones Docker & Podman provide. System containers being important here, I don't so much need immutable build once run anywhere systems, but rather an isolated environment in which I can pile on old dependencies, side loaded tooling, etc. All of it accessible from every system I use with no more than a simple incus shell saltext-copier call.

And incus makes it incredible easy to define and build custom containers! All you really need to do is have a rootfs to work with, and after that just a little bit of yaml will let you build the necessary squashfs with distrobuilder.

Here's the base definition for my Alpine 3.17 based Saltext container. Feel free to build and use it yourself.

image:
  distribution: alpine
  release: v3.17
  architecture: x86_64

source:
  downloader: rootfs-http
  url: https://dl-cdn.alpinelinux.org/alpine/v3.17/releases/x86_64/alpine-minirootfs-3.17.9-x86_64.tar.gz

packages:
  manager: apk
  sets:
    - packages:
      - gcc
      - python3
      - py3-pip
      - python3-dev
      - musl-dev
      - linux-headers
      - dhcpcd
      - openrc
      - mg
      - git
      - bash
      - openssh
      - openssh-keygen
      action: install

actions:
  - trigger: post-packages
    action: |-
      #!/bin/sh
      rc-update add networking boot
      rc-update add dhcpcd boot

  - trigger: post-packages
    action: |-
      #!/bin/sh
      pip3 install copier --ignore-installed packaging
      pip3 install copier_templates_extensions --ignore-installed packaging
      
  - trigger: post-packages
    action: |-
      #!/bin/sh
      # Create basic network interface config
      cat > /etc/network/interfaces << EOF
      auto lo
      iface lo inet loopback
      
      auto eth0
      iface eth0 inet dhcp
      EOF

targets:
  incus: {}

Which is seriously as easy as running a single command.

sudo distrobuilder build-incus alpine-3.17.yaml

Which will produce two files, a squashfs and some incus metadata, which we can then use to import the image into our cluster. We then only need to launch a container from our built Alpine 3.17 base and it will pop up ready to go, with most of the base tooling ready to use!

incus image import incus.tar.xc rootfs.squashfs --alias alpine-3.17-saltext
incus launch alpine-3.17-saltext saltext-copier
incus config device add saltext-copier shm disk source=/dev/shm path=/dev/shm

Of course, I still needed to do some small quality of life things to this container, like configure git and add Emacs. It was only meant to be a temporary base system until we migrate away from 3.10 and to something more modern. Though chances are that the LTS release will continue to lag behind and I'll be re-creating more fleshed out custom images in the future.

Why didn't you just Cloud Init?

Great question! I would normally just use a cloud enabled incus image (images:alpine/edge/cloud) and provide a custom configuration to the container via cloud-init. I do this extensively for testing Ansible playbooks, or deploying things in my homelab. It just makes it easier to distribute things like SSH keys, etc, that you'd want configured during launch time. Sort of the expectation these days with AWS et al.

But, these images only go back as far as Alpine 3.19 in Incus. We stopped packaging Python 3.10 in v3.17.5 so that's the last version of Alpine I can pull down to get access to the right ecosystem of tools for this job.

My hope is I can maybe simplify this in the future once we're up to speed with 3.12 or 3.13 if it takes that long.

Additionally, I could have compiled Python from source and maintained a 3.10 version in personal APK repo. But that sounds like even more effort than I've already put into this, and I literally need it for like 2-3 tools that I know will be upgraded alongside Salt itself. We're going for the solution with the most bang for our buck here.

Actually migrating s3fs

All of this set the stage to finally migrate a few modules. I chose to start with s3fs for two reasons; I'm really reliant on this because I don't want to stash non-configuration data in my salt master, and it's literally a single python file. That's about as gentle an introduction to this as one could hope for. I just have to move one file, how hard could it be?

Well, after about 2 nights of effort, I can tell you that the gut reaction is to reach for saltext-copier as indicated in the name of the container. But that my friend is the hard path. It turns out that for the deprecated modules a custom migration tool was written around the copier tool which helps preserve git history and automatically updates non-compliant code.

Installing this tool is a simple uv command away, and that enables us to simply tell the migration tool the virtual name of the module we need to migrate.

uv tool install --python 3.10 git+https://github.com/salt-extensions/salt-extension-migrate
saltext-migrate s3fs

Once you kick off the migration tool it'll walk you through selecting the files you want to migrate, in this case it was literally just modules/fileserver/s3fs.py. And then begin skimming the salt git repo for history to preserve. It's actually a really neat process that materially lowers the barrier to entry. It's literally as simple as following through the prompts the copier and migration tools provide, which upon completion will drop you into a brand new structure git repo and provide you with curated next steps based on the code base you moved.

For example, this is the pre-commit linting information I got when initially attempting to pull out the apk package module from the larger pkg virtual environment. All super easy and actionable and small things to change.

Pre-commit is failing. Please fix all (2) failing hooks

✗ Failing hook (1): Check CLI examples on execution modules
- hook id: check-cli-examples
- exit code: 1

The function 'purge' on 'src/saltext/pkg/modules/apkpkg.py' does not have a 'CLI Example:' in it's docstring

✗ Failing hook (2): Lint Source Code
- hook id: nox
- exit code: 1

nox > Running session lint-code-pre-commit
nox > python -m pip install --progress-bar=off wheel
nox > python -m pip install '.[lint,tests]'
nox > pylint --rcfile=.pylintrc --disable=I src/saltext/pkg/modules/apkpkg.py
************* Module pkg.modules.apkpkg
src/saltext/pkg/modules/apkpkg.py:141:7: R1729: Use a generator instead 'any(salt.utils.data.is_true(kwargs.get(x)) for x in ('removed', 'purge_desired'))' (use-a-generator)
src/saltext/pkg/modules/apkpkg.py:207:4: C0206: Consider iterating with .items() (consider-using-dict-items)
src/saltext/pkg/modules/apkpkg.py:486:4: R1720: Unnecessary "else" after "raise", remove the "else" and de-indent the code inside it (no-else-raise)
src/saltext/pkg/modules/apkpkg.py:550:12: R1724: Unnecessary "else" after "continue", remove the "else" and de-indent the code inside it (no-else-continue)

------------------------------------------------------------------
Your code has been rated at 9.82/10 (previous run: 9.82/10, +0.00)

nox > Command pylint --rcfile=.pylintrc --disable=I src/saltext/pkg/modules/apkpkg.py failed with exit code 24
nox > Session lint-code-pre-commit failed.

So now what?

Well with the tooling in the right place, and an issue opened to officially adopt s3fs, I should probably talk to someone about how best to migrate the apk pkg module since it's a small subset of a much large set of extensions. I might accidentally break something if I guess wrong.

I also need to figure out how best to turn these extensions into Alpine packages and contribute them to Aports alongside my salt packages. My end goal will be to have a nice simple salt docker container where I can just do something like this:

FROM registry.alpinelinux.org/img/alpine:edge

RUN <<EOF
apk add --no-cache salt salt-master saltext-s3fs saltext-apk
rm -rf /var/cache/apk/*
rm -rf /tmp/*
EOF

ENTRYPOINT ["salt-master"]

If any of this seems interesting and you want to learn more about the extension system in Salt, I'm finding that Extending Salt by Joseph Hall to be a phenomenal resource. I'm chewing through it currently to get an even better grasp on how Salt's internals work while I work through all of this.