(λ (x) (create x) '(knowledge))

Packaging Private Projects

Gitea, abuild, and lots of Lisp ยท July 16th, 2024

This is one I've been sitting on for a minute. I've been doing some dev work on an old repo of mine, a little passion project that a friend and I have been working on for years on and off. As you can imagine, while I would happily open source the world, my friend has a 50% say in what happens with that code base after all the time we've put into it. So it lives in a private Gitea instance, under a private group only we can access.

It makes me sad I can't just link to it, because it's lots of neat Common Lisp things that just tickle all of the parts of my brain that make me happy. Maybe one day I'll convince him.

Regardless, it's a private repo, locked away tight.

I picked that project back up recently because I wanted to know what it would take to get from where we left off to an MVP product. What's realistically left to be done and how much of a lift in development time would it be to create even the most bare bones of prototype. So naturally I started to dig into all of my various Makefiles, what dependencies do I need to have on a system, what carch were we targeting and all of that, so I could hopefully wrap these little packages into a nice little APKBUILD.

One small problem to rain on my day, abuild can only fetch publicly available endpoints. Shame right? I can't reasonably just make the repos publicly visible, that would make it possible to package, but it'd probably irritate Kevin a bit.

Fortunately, Gitea has me covered

Gitea is this awesome little git forge that has so many neat little tricks up its sleeve. One of those is authentication tokens that can be included as part of any GET request.

This means that in our APKBUILD we can do something like this

source="https://example.com/user/repo/archive/$_commit.tar.gz?token=$our_gitea_token"

And then fetch the private repository as though it was publicly available! Now it would suck to lean credentials in any sort of build script, fortunately APKBUILDs are essentially just shell scripts that get read in and executed, which means if you use a CLI password manager like pass, lpass, or rbw then you can just embed a call to your program in lieu of including the token.

Here's an example of what I mean, this is the APKBUILD file for our drive thru event engine.

# Contributor: Will Sinatra <wpsinatra@gmail.com>
# Maintainer: Will Sinatra <wpsinatra@gmail.com>
pkgname=kw-event
pkgver=0.1_git20240626
_commit=9a3b944349b7cb12af2168d0126848db0a7ba20c
pkgrel=0
pkgdesc="KlockWork Systems Event Engine"
url="https://krei.lambdacreate.com/kws/kw-event"
arch="x86_64 aarch64"
license="custom"
makedepends="sbcl sqlite-dev"
depends="sqlite"
builddir="$srcdir/$pkgname"
subpackages="$pkgname-openrc"
options="!check net !strip"

# We have submodule dependencies, so we need to pull the git hash for each of those
_common_rev="361fbeb3df404a6480d0b644f82502acbfdc712b"
_submodules="kw-common-$_common_rev"

# And then add them to the sources as well
source="$pkgname-$pkgver.tar.gz::https://krei.lambdacreate.com/kws/kw-event/archive/$_commit.tar.gz?token=$(pass show kws/api/krei)
	kw-common-$_common_rev.tar.gz::https://krei.lambdacreate.com/kws/kw-common/archive/$_common_rev.tar.gz?token=$(pass show kws/api/krei)
	kw-event.post-install"

prepare() {
	default_prepare
	cd "$srcdir"
	
	_vendor_dir="$srcdir"/kw-event/src/

	# split library-commit into just library name
	for _mod in $_submodules; do
		_mod_rev=${_mod##*-}
		_mod_name=$(echo $_mod | sed -e "s/-$_mod_rev//")
		mv $_mod_name $_vendor_dir
	done
}

build() {
	make all
}

package() {
	  #install -Dm644 "$builddir"/kw-event.cfg "$pkgdir"/etc/kws/kw-event.cfg

	  install -Dm755 "$builddir"/init.d/kw-event "$pkgdir"/etc/init.d/kw-event
	  install -Dm755 "$builddir"/src/kw-event "$pkgdir"/usr/bin/kw-event
}

sha512sums="
8c9799b03acda4e2481838c022a846d956bad5244034b5ddd507033ef914b260aa970e598c8d4ab053b0cf75ec69c9e5e8f65e4d3a5b572cdb3cf2d5d829f4aa  kw-event-0.1_git20240626.tar.gz
d13e06ade963207897893c3e80e21e9a82befbf674b05f7c93ead5a4495c40c5fce282e6bc27b5192c7b2bd2f45a7c0b3d83635f4e1428e936a78111118bf9ab  kw-common-361fbeb3df404a6480d0b644f82502acbfdc712b.tar.gz
b0d20e3482c1ca878281327f6fcb1881e00d14e8272a34e2dc317c4e92539f5f17c5f337381ab4150aaa71141236ea131676463a4ab5df2d819acdb75866c546  kw-event.post-install
"

The result of this little find is that I can happily keep these repositories private until they no longer need to be, but I can continue to package and distribute the software in a way that is extremely familiar to me, and that makes getting a prototype device together that much easier.

In other news

OCC really curtailed my ability to work on this particular project unfortunately. You'll notice in the APKBUILD that I'm only targeting x86 and arm64 hardware, that's because those were the two carches we really did any serious development on.

Because of that, I didn't remember until a few days ago that SBCL hasn't properly built on Alpine x86 since December of 2023, that's version 2.3.11, and we're on 2.4.6 currently.

I initially thought that this is no big deal, I maintain SBCL! I have all the tools I need to get this fixed and working, I'll just run it through abuild -r real quick, and then I'll do more lisp dev and be a happy OCC camper.

But 4 hours into compiling SBCL on my netbook I get this amazing error.

CORRUPTION WARNING in SBCL pid 23303 tid 23303:
Memory fault at 0 (pc=0x92222f8 [code 0x9222000+0x2F8 ID 0x92a], fp=0xb7ac7bac, sp=0xb7ac7b98) tid 23303
The integrity of this image is possibly compromised.
Exiting.

Certainly that's caused by our CFLAGS or something dumb, so I just went ahead and compiled it by hand. Except another 4 hours later, I get the same error message..

Now I tried this several more times, my little Acer ZG5 is no speed demon. Over the last 3 days I pretty much just let the poor thing roast itself trying to compile, debug, compile, debug. And I never really got anywhere further with it, despite burning around 24hrs of compilation time.

Fortunately, ldb provides a neat backtrace functionality so you can see where the compilation process fails. And this is enough information to file a bug report upstream, which is exactly what I've done.

Welcome to LDB, a low-level debugger for the Lisp runtime environment.
ldb> ba
Backtrace:
    0: fp=0xb7ac7bac pc=0x92222f8 SB-IMPL::GET-INFO-VALUE
    1: fp=0xb7ac7bf4 pc=0x91d2a32 SB-C::CHECK-VARIABLE-NAME-FOR-BINDING
    2: fp=0xb7ac7c50 pc=0x91bf9fa (FLET SB-C::CHECK-NAME :IN SB-INT::CHECK-LAMBDA-LIST-NAMES)
    3: fp=0xb7ac7cac pc=0x91bf652 SB-INT::CHECK-LAMBDA-LIST-NAMES
    4: fp=0xb7ac7d80 pc=0x9948e85 SB-C::IR1-CONVERT-LAMBDA
    5: fp=0xb7ac7da4 pc=0x99211c5 SB-C::IR1-TOPLEVEL-FOR-COMPILE
    6: fp=0xb7ac7dc8 pc=0x9921706 SB-C::%COMPILE
    7: fp=0xb7ac7e18 pc=0x99229cf (LAMBDA () :IN SB-C::COMPILE-IN-LEXENV)
    8: fp=0xb7ac7e80 pc=0x98fba8d (FLET SB-C::WITH-IT :IN SB-C::%WITH-COMPILATION-UNIT)
    9: fp=0xb7ac7ef8 pc=0x99221d9 SB-C::COMPILE-IN-LEXENV
   10: fp=0xb7ac7f20 pc=0x99233f6 COMPILE
   11: fp=0xb7ac7f30 pc=0xa26f45d SB-PRETTY::!PPRINT-COLD-INIT
   12: fp=0xb7ac7f80 pc=0xa4d9e8e SB-KERNEL::!COLD-INIT

So I guess OCC this year has really been Alpine maintenance focused. But at least there's something tangible because of it! All of those Ruby packages were merged, so I bet we'll get an SBCL fix in short order, though maybe not before the end of the OCC.