(λ (x) (create x) '(knowledge))

Random Script: maintained.sh

How I manage package management ยท October 11th, 2023

As most everyone knows, I'm a package maintainer for my favorite Linux distribution, Alpine. I use Alpine everywhere, on my droid, in my containers, on my NAS and LXD clusters. It even powers the web server that runs this blog! I really like Alpine. And for all that joy I get using it, I get to give back by contributing to the packages available in the repos! It's kind of great if you ask me. When I started maintaining 5 years ago I really only thought I'd manage SBCL. Maybe a few random packages here and there, but nothing big. I just need that one language to work on arm for a software project a friend and I were working on together. And boy was it difficult! I remember being extremely frustrated that the package was dropped. It couldn't build on the arches I needed, and it was barely building on easier ones like x86_64. At that time 5 years ago it felt amazing to not only get the build working, but to take ownership of the process. Of course, neither my friend nor I really understood what it was like to contribute to FOSS, a fact that is aptly captured in this meme he made to celebrate my victory over SBCL.

A crass meme my friend Kevin made to express his impression of me packaging SBCL.

Back then both of us were used to dealing with monolithic companies and organizations that not only gave no shits about the end user experience, but also were often times hostile to people poking around in their territory. For us, it felt like SBCL being removed right after we started using it was Alpine fucking things up. But in reality it's the natural progress of packaging in a Linux distribution. If nobody steps up to the plate and takes responsibility for the health and maintenance of the package, it's going to get dropped. And when that package is broken to the point where it won't build, it rapidly goes from maintained to unmaintained. That isn't the distro's fault, and it's why all of our continued volunteer work is so important. I'm glad I took the opportunity to invest myself in the Alpine ecosystem, solving this one frustrating issue has blossomed to so much more.

While the project that sparked all of this is long since abandoned the packaging has kept on. As of today I maintain 67 different packages, and have enough in the works to readily push it past 80 once I get all of my dependencies sorted out! That's an exciting number if you ask me! But it's also a lot of packages, and life can get pretty busy when you've got a 7 year old and another kid on the way. I realistically only manage all of these packages with the support of the Alpine packaging community as a whole. There are countless times where I've missed an update and someone else has made the MR for it. Or something will break, a new feature is needed, and someone who has never touched an alpine package before make a contribution and solves a problem I wasn't aware of. It's beautiful, and analogues my own experiences so much!

But that's why I'm writing this, I don't plan to stop any time soon, and something I realized during the OCC this year is that I don't have a particularly portable method to track my contributions. My alerting is all kind of "when I look" based. I don't even follow all of the RSS feeds for all the packages I maintain. It's kind of rough. But I'm a sysadmin damn it, I can write a shell script to solve this problem! And so I did precisely that, and present to you my Rube Goldberg machine of a maintenance tracking system. It relies heavily on my aports fork, recutils, and absolutely eschewing good practice and just EOFing python into the python interpreter so I can deal with as little of that horrid language as is humanly possible. I love this script none the less though, for it gives me the system I so sorely lack in maintaining all these things.

#!/bin/ash
#Create and maintain a database of info to help manage alpine packages
repo=/home/durrendal/Development/aports
apkcache=/var/tmp/apks.txt
apkpersist=/var/tmp/apks.rec
email="wpsinatra@gmail.com"

#colors
red='\e[31m'
green='\e[32m'
yellow='\e[33m'
blue='\e[34m'
reset='\e[0m'

#Install necessary packages
setup() {
	apk add gawk sed grep coreutils recutils git python3 py3-feedparser ripgrep
}

#Force update git repo
update_repo() {
	cd $repo
	if ["$(git branch | grep '^\*' | sed 's/\* //')" != "master"]; then
		git switch branch master
	fi
	git fetch upstream
	git rebase upstream/master
	git push
}

#clunky releases rss/atom feed pattern mattern. This has like a 50% success so far
rss_feed() {
	#https://github.com/:owner/:repo/releases.atom
	#https://gitlab.com/:owner/:repo/-/tags?format=atom
	#https://git.sr.ht/~:owner/:repo/refs/rss.xml
	#https://codeberg.org/:owner/:repo.rss

	if [$(echo $1 | grep -o "github")]; then
		echo "$1/releases.atom"
	elif [$(echo $1 | grep -o "gitlab")]; then
		echo "$1/-/tags?format=atom"
	elif [$(echo $1 | grep -o "git.sr.ht")]; then
		echo "$1/refs/rss.xml"
	elif [$(echo $1 | grep -o "codeberg")]; then
		echo "$1/releases.rss"
	fi
}

#Query repology for reported outdated packages
outdated_apks() {
	python3 - <<EOF
import requests, json, urllib
from urllib.request import urlopen

api = "https://repology.org/api/v1/"
data = urlopen(api + "/projects/?inrepo=alpine_edge&maintainer=$email&outdated=1")
json_object = json.load(data)
pkgs = []

for pkg in json_object:
    pkgs.append(pkg)

for pkg in pkgs:
    print(pkg)
EOF
}

#Pull an RSS feed and check the title of the first post for a version number
check_feed() {
	title=$(python3 - <<EOF
import feedparser
feed = feedparser.parse("$1")
entry = feed.entries[0]
print(entry.title)
EOF
		)
	echo "$title" | sed 's/'$pkg'//g' | grep -m1 -Eo "([0-9]+)((\.)[0-9]+)*[a-z]*" | head -n1
}

#Check all feeds for version changes
check_feeds() {
	for pkg in $(recsel -C -P name $apkpersist); do
		feed=$(recsel -e "name = '"$pkg"'" -P rss $apkpersist)
		pver=$(recsel -e "name = '"$pkg"'" -P version $apkpersist)
		if ["$feed" != ""]; then
			#If the feed is invalid you'll get traceback errors.
			rver=$(check_feed "$feed")

			if ["$1" != "version_only"]; then
				if ["$rver" == ""]; then
					rver=0
				fi
				
				if $(awk -v a="$pver" -v b="$rver" 'BEGIN{if (a >= b) exit 0; else exit 1}'); then
					pver="$green$pver$reset"
				elif $(awk -v a="$pver" -v b="$rver" 'BEGIN{if (a < b) exit 0; else exit 1}'); then
					rver="$red$rver$reset"
				fi
				
				echo -e "$pkg $pver $rver"
			fi
		fi
	done
}

#Generate a status list
list_apks() {
	if ["$1" == "all"]; then
		for pkg in $(recsel -C -P name $apkpersist); do
			pver=$(recsel -e "name = '"$pkg"'" -P version $apkpersist)
			rver=$(recsel -e "name = '"$pkg"'" -P rssv $apkpersist)
			flagged=$(recsel -e "name = '"$pkg"'" -P flagged $apkpersist)
			rss=$(recsel -e "name = '"$pkg"'" -P rss $apkpersist)
			
			#If outdated display in red, otherwise green
			if ["$flagged" == "yes"]; then
				pver="$red[R]$pver$reset"
			elif $(awk -v a="$pver" -v b="$rver" 'BEGIN{if (a <= b) exit 0; else exit 1}'); then
				pver="$red$pver$reset"
			else
				pver="$green$pver$reset"
			fi
			
			#if the DB is missing an RSS feed, report yellow
			if ["$rss" == ""]; then
				pkg="$yellow$pkg$reset"
			else
				pkg="$pkg$reset"
			fi
			
			#if the rss version is 000, report yellow
			if ["$rver" == "000"]; then
				rver="$yellow$rver$reset"
			fi
			
			echo -e "$pkg $pver $rver"
		done
	else
		for pkg in $(recsel -C -e "suppressed != '"true"'" -P name $apkpersist); do
			pver=$(recsel -e "name = '"$pkg"'" -P version $apkpersist)
			rver=$(recsel -e "name = '"$pkg"'" -P rssv $apkpersist)
			flagged=$(recsel -e "name = '"$pkg"'" -P flagged $apkpersist)
			rss=$(recsel -e "name = '"$pkg"'" -P rss $apkpersist)

			if $(awk -v a="$pver" -v b="$rver" 'BEGIN{if (a < b) exit 0; else exit 1}'); then
				pver="$red$pver$reset"
				echo -e "$pkg $pver $rver"
			elif ["$flagged" == "yes"]; then
				pver="$red[R]$pver$reset"
				echo -e "$pkg $pver $rver"
			fi
		done
	fi
}

count_apks() {
	count=$(recsel -C -P name $apkpersist | wc -l)
	echo "Maintained: $count"
}

find_abandoned() {
	apk_paths=$(rg -H -N "^# Maintainer:( )*$" $repo | sed 's/\n//g' | awk -F':#' '{print $1}')
	for apk in $apk_paths; do
		repo=$(echo $apk | grep -Eo 'testing.*|community.*|main.*' | awk -F'/APKBUILD' '{print $1}')
		desc=$(grep -m1 pkgdesc $apk | awk -F'=' '{print $2}')
		echo -e "$blue$repo$reset $desc"
	done
}

find_maintainers_packages() {
	#if we pass an email directly, simple search for it
	if ["$(echo "$1" | grep -o "@")" == "@"]; then
		maintainer="$1"
	#Check to see if $1 is a file in aports, resolve the maintainer, and return all packages
	else
		file="$(rg --files $repo | rg --word-regexp "$1" | grep APKBUILD)"
		for f in $file; do
			if [-f $f]; then
				maintainer="$(cat $f | grep "^# Maintainer:" | awk -F':.*<' '{print $2}' | sed 's/>//g')"
			else
				echo "$1 cannot be found, try searching for the maintainer's email or a different package."
				exit 1
			fi
		done
	fi

	if ["$maintainer" != ""]; then
		apk_paths=$(rg -H -N "^# Maintainer:.*<$maintainer>$" $repo | sed 's/\n//g' | awk -F':#' '{print $1}')
		for apk in $apk_paths; do
			repo=$(echo $apk | grep -Eo 'testing.*|community.*|main.*' | awk -F'/APKBUILD' '{print $1}')
			desc=$(grep -m1 pkgdesc $apk | awk -F'=' '{print $2}')
			echo -e "$blue$repo$reset $desc"
		done
	else
		echo "Maintainer not found"
	fi
}

#List the RSS address of all packages
list_rss() {
	if ["$1" == "urls"]; then
		for feed in $(recsel -C -P rss $apkpersist); do
			if ["$feed" != ""]; then
				echo $feed
			fi
		done
	elif ["$1" == "empty"]; then
		recsel -C -P name -e "rss = ''" $apkpersist
	fi
}

#Reconcile DB with changes from git
rec_reconcile () {
	if [! -f $apkpersist]; then
		cat > $apkpersist <<EOF
%rec: apk
%doc: APK Info

%key: name
%type: name string
%type: path string
%type: version string
%type: rssv string
%type: flagged string
%type: rss string
%type: suppressed string
%type: updated_on: date

%mandatory: name version
%allowed: name version rssv flagged rss updated_at
%auto: updated_on

EOF
	else
		cp $apkpersist /tmp/apks-$(date +%Y-%m-%d-%H%M%S).rec
	fi

	#Ensure we're not reconsiling on the wrong branch
	cd $repo
	if ["$(git branch | grep '^\*' | sed 's/\* //')" != "master"]; then
		echo "Aports isn't on the master branch currently, please switch before reconciling."
		exit 1
	fi

	#Grep the entire locally cloned aports repo for the defined maintainer, then create a csv file of import info for easier parsing.
	#apk_paths=$(grep -H -r $email $repo | grep Maintainer | awk -F':#' '{print $1}') #this takes 2m
	apk_paths=$(rg -H -N "^# Maintainer:.*<$email>" $repo | sed 's/\n//g' | awk -F':#' '{print $1}') #this takes 3s
	for apk in $apk_paths; do
		path=$(echo $apk)
		apks=$(echo $path | awk -F'/APKBUILD' '{print $1}' | sed 's|'$repo'/||')
		name=$(echo $apks | awk -F'/' '{print $2}')
		pver=$(cat $path | grep "^pkgver=" | awk -F'=' '{print $2}')

		#Attempt to resolve RSS path, if it doesn't exist in the db
		crss=$(recsel -e "name = '"$name"'" -P rss $apkpersist)
		if ["$crss" == ""]; then
			src=$(cat $path | grep "^source=" | awk -F'=' '{print $2}' | grep -Eo "(http|https)://.*+" | tr -d \'\")
			url=$(cat $path | grep "^url=" | awk -F'=' '{print $2}' | grep -Eo "(http|https)://.*+" | tr -d \'\")
			#If the url ends in some trailing .extension, ie: .xz, .gz, .zip
			if [$(echo $url | grep -Eo "\.[a-z]+$") ]; then
				#Try to make a valid RSS feed. This'll fail if the project name is different from package name (ie: ChezScheme is chez-scheme)
				src_url=$(echo $src | sed 's|'$name'/.*|'$name'|')
				rss=$(rss_feed $src_url)
			else
				rss=$(rss_feed $url)
			fi
		else
			rss=$crss
		fi
		rdate=$(date)
		
		if ["$(recsel -e "name = '"$name"'" -P name $apkpersist)" == "$name"]; then
			recset -e "name = '"$name"'" -t apk -f path -s "$path" --verbose $apkpersist
			recset -e "name = '"$name"'" -t apk -f version -s "$pver" --verbose $apkpersist
			recset -e "name = '"$name"'" -t apk -f updated_on -s "$rdate" --verbose $apkpersist
		else
			recins -t apk -f name -v "$name" -f path -v "$path" -f version -v "$pver" -f rssv -v "0" -f flagged -v "no" -f rss -v "$rss" -f updated_on -v "$rdate" --verbose $apkpersist
		fi
	done
}

#Skim configured RSS feeds and repology api for outdated packages
check_outdated() {
	outdated=$(outdated_apks | sed 's/:/-/g')
	for pkg in $(recsel -C -P name $apkpersist); do
		feed=$(recsel -e "name = '"$pkg"'" -P rss $apkpersist)
		flagged=$(echo "$outdated" | grep -o "$pkg")
		rssv=000
		if ["$feed" != ""]; then
			rssv=$(check_feed "$feed")
		fi

		echo "$pkg feed reports version $rssv"
		recset -t apk -e "name = '"$pkg"'" -f rssv -s "$rssv" --verbose $apkpersist
		
		if ["$flagged" == "$pkg"]; then
			echo "$pkg has been flagged outdated on repology"
			recset -t apk -e "name = '"$pkg"'" -f flagged -s "yes" --verbose $apkpersist
		elif ["$flagged" != "$pkg"]; then
			recset -t apk -e "name = '"$pkg"'" -f flagged -s "no" --verbose $apkpersist
		fi
	done
}

if ["$1" == "-l"]; then
	list_apks $2
elif ["$1" == "-lf"]; then
	if [-z $2]; then
		echo "list_rss requires either urls or empty as an argument"
		exit 1
	fi
	list_rss $2
elif ["$1" == "-cv"]; then
	check_outdated
elif ["$1" == "-c"]; then
	count_apks
elif ["$1" == "-fa"]; then
	find_abandoned
elif ["$1" == "-fm"]; then
	if [-z $2]; then
		echo "find_maintainers requires either a package name or maintainer email"
		exit 1
	fi
	find_maintainers_packages $2
elif ["$1" == "-r"]; then
	rec_reconcile
elif ["$1" == "-ur"]; then
	update_repo
elif ["$1" == "-o"]; then
	outdated_apks
else
	printf 'Usage: maintained.sh [-l] [-lf] [-c] [-cv] [-ur] [-r]
-l [all] | List packages & version. If red, out of date. If yellow, missing rss
-lf [urls|empty] | List package available or unconfigured rss feeds
-c | List total number of maintained apks
-fa | Find abandoned packages
-fm [email|package] | Find packages maintained by someone else
-cv | Check rss feeds & Repology for latest package version & cache
-r | Reconcile persistent recstore against aports
-ur | update aports fork master branch
'
	exit 0
fi

Yes the script is a little janky, there's definitely a better way to do the python stuff, in fact the entire thing could probably be a python script, or nim, or go. Lets be honest, I really wanted a shell script. The difference in my mind is that this isn't a tool I'm writing to put into a repo to be consumed publicly. It is the horrible glue that keeps some thing together. That thing is my sanity maintaining so many packages. Shell glue is allowed to be ugly, as long as it's functional and reasonably easy to extend/maintain. I think this fits aptly! But it probably doesn't mean too terribly much to you if I don't explain what it's all for.

maintained.sh's single most important purpose in life is the generation of a rec file database that tracks information about all my packages. This allows me to do simple select style queries to determine if a package is out of date, or where it lives, or even just how many of these things I own the responsibility of. It is not proactive, but it is simple enough that I can work it into my daily workflow to constantly stay on top of!

The recfile that the script generates looks like lots of little blocks like this, simple plaintext.

name: sbcl
path: /home/durrendal/Development/aports/community/sbcl/APKBUILD
version: 2.3.9
rssv: 2.3.9
flagged: no
rss: https://github.com/sbcl/sbcl/releases.atom
updated_on: Mon Oct 9 09:37:37 EDT 2023

This is important for several reasons. 1) recutils is awesome, 2) it's plain text so if I outgrow it I sed/grep/awk it into insert queries and migrate to sqlite3, 3) did I mention recutils is awesome, that's because it's a whole bespoke plain text database system and if it didn't exist I'd probably try and do this using an org file. I can just quickly read this if I need it, I can parse it however I want. Perfect glue level functionality.

In essence this is just a love letter to the *nix design. Simple ubiquitous text interfaces everywhere, so that you can string together simple tools into custom solutions. I absolutely adore this, and apply it everywhere, from packaging to photography, it makes life so much more enjoyable.

Note:

I initially wrote this post in early August, during a fervor of productivity in the Alpine packaging realm. Then life got incredibly busy again, and I failed to ever publish this post even though it was essentially done save for spell checking. Sometimes "How I keep up with my packages" actually means I don't. As ingenuitive as I can be, I would not be able to manage any of this without the hard work and continuous assistance of other Alpine packagers, a sincere thank you goes out to anyone who's bumped anything I maintain!

Bio

(defparameter *Will_Sinatra* '((Age . 31) (Occupation . DevOps Engineer) (FOSS-Dev . true) (Locale . Maine) (Languages . ("Lisp" "Fennel" "Lua" "Go" "Nim")) (Certs . ("LFCS"))))

"Very little indeed is needed to live a happy life." - Aurelius