Random Script: maintained.sh
How I manage package management · October 11th, 2023
As most everyone knows, I'm a package maintainer for my favorite Linux distribution, Alpine. I use Alpine everywhere, on my droid, in my containers, on my NAS and LXD clusters. It even powers the web server that runs this blog! I really like Alpine. And for all that joy I get using it, I get to give back by contributing to the packages available in the repos! It's kind of great if you ask me. When I started maintaining 5 years ago I really only thought I'd manage SBCL. Maybe a few random packages here and there, but nothing big. I just need that one language to work on arm for a software project a friend and I were working on together. And boy was it difficult! I remember being extremely frustrated that the package was dropped. It couldn't build on the arches I needed, and it was barely building on easier ones like x86_64. At that time 5 years ago it felt amazing to not only get the build working, but to take ownership of the process. Of course, neither my friend nor I really understood what it was like to contribute to FOSS, a fact that is aptly captured in this meme he made to celebrate my victory over SBCL.
Back then both of us were used to dealing with monolithic companies and organizations that not only gave no shits about the end user experience, but also were often times hostile to people poking around in their territory. For us, it felt like SBCL being removed right after we started using it was Alpine fucking things up. But in reality it's the natural progress of packaging in a Linux distribution. If nobody steps up to the plate and takes responsibility for the health and maintenance of the package, it's going to get dropped. And when that package is broken to the point where it won't build, it rapidly goes from maintained to unmaintained. That isn't the distro's fault, and it's why all of our continued volunteer work is so important. I'm glad I took the opportunity to invest myself in the Alpine ecosystem, solving this one frustrating issue has blossomed to so much more.
While the project that sparked all of this is long since abandoned the packaging has kept on. As of today I maintain 67 different packages, and have enough in the works to readily push it past 80 once I get all of my dependencies sorted out! That's an exciting number if you ask me! But it's also a lot of packages, and life can get pretty busy when you've got a 7 year old and another kid on the way. I realistically only manage all of these packages with the support of the Alpine packaging community as a whole. There are countless times where I've missed an update and someone else has made the MR for it. Or something will break, a new feature is needed, and someone who has never touched an alpine package before make a contribution and solves a problem I wasn't aware of. It's beautiful, and analogues my own experiences so much!
But that's why I'm writing this, I don't plan to stop any time soon, and something I realized during the OCC this year is that I don't have a particularly portable method to track my contributions. My alerting is all kind of "when I look" based. I don't even follow all of the RSS feeds for all the packages I maintain. It's kind of rough. But I'm a sysadmin damn it, I can write a shell script to solve this problem! And so I did precisely that, and present to you my Rube Goldberg machine of a maintenance tracking system. It relies heavily on my aports fork, recutils, and absolutely eschewing good practice and just EOFing python into the python interpreter so I can deal with as little of that horrid language as is humanly possible. I love this script none the less though, for it gives me the system I so sorely lack in maintaining all these things.
#!/bin/ash
#Create and maintain a database of info to help manage alpine packages
repo=/home/durrendal/Development/aports
apkcache=/var/tmp/apks.txt
apkpersist=/var/tmp/apks.rec
email="wpsinatra@gmail.com"
#colors
red='\e[31m'
green='\e[32m'
yellow='\e[33m'
blue='\e[34m'
reset='\e[0m'
#Install necessary packages
setup() {
apk add gawk sed grep coreutils recutils git python3 py3-feedparser ripgrep
}
#Force update git repo
update_repo() {
cd $repo
if [ "$(git branch | grep '^\*' | sed 's/\* //')" != "master" ]; then
git switch branch master
fi
git fetch upstream
git rebase upstream/master
git push
}
#clunky releases rss/atom feed pattern mattern. This has like a 50% success so far
rss_feed() {
#https://github.com/:owner/:repo/releases.atom
#https://gitlab.com/:owner/:repo/-/tags?format=atom
#https://git.sr.ht/~:owner/:repo/refs/rss.xml
#https://codeberg.org/:owner/:repo.rss
if [ $(echo $1 | grep -o "github") ]; then
echo "$1/releases.atom"
elif [ $(echo $1 | grep -o "gitlab") ]; then
echo "$1/-/tags?format=atom"
elif [ $(echo $1 | grep -o "git.sr.ht") ]; then
echo "$1/refs/rss.xml"
elif [ $(echo $1 | grep -o "codeberg") ]; then
echo "$1/releases.rss"
fi
}
#Query repology for reported outdated packages
outdated_apks() {
python3 - <<EOF
import requests, json, urllib
from urllib.request import urlopen
api = "https://repology.org/api/v1/"
data = urlopen(api + "/projects/?inrepo=alpine_edge&maintainer=$email&outdated=1")
json_object = json.load(data)
pkgs = []
for pkg in json_object:
pkgs.append(pkg)
for pkg in pkgs:
print(pkg)
EOF
}
#Pull an RSS feed and check the title of the first post for a version number
check_feed() {
title=$(python3 - <<EOF
import feedparser
feed = feedparser.parse("$1")
entry = feed.entries[0]
print(entry.title)
EOF
)
echo "$title" | sed 's/'$pkg'//g' | grep -m1 -Eo "([0-9]+)((\.)[0-9]+)*[a-z]*" | head -n1
}
#Check all feeds for version changes
check_feeds() {
for pkg in $(recsel -C -P name $apkpersist); do
feed=$(recsel -e "name = '"$pkg"'" -P rss $apkpersist)
pver=$(recsel -e "name = '"$pkg"'" -P version $apkpersist)
if [ "$feed" != "" ]; then
#If the feed is invalid you'll get traceback errors.
rver=$(check_feed "$feed")
if [ "$1" != "version_only" ]; then
if [ "$rver" == "" ]; then
rver=0
fi
if $(awk -v a="$pver" -v b="$rver" 'BEGIN{if (a >= b) exit 0; else exit 1}'); then
pver="$green$pver$reset"
elif $(awk -v a="$pver" -v b="$rver" 'BEGIN{if (a < b) exit 0; else exit 1}'); then
rver="$red$rver$reset"
fi
echo -e "$pkg $pver $rver"
fi
fi
done
}
#Generate a status list
list_apks() {
if [ "$1" == "all" ]; then
for pkg in $(recsel -C -P name $apkpersist); do
pver=$(recsel -e "name = '"$pkg"'" -P version $apkpersist)
rver=$(recsel -e "name = '"$pkg"'" -P rssv $apkpersist)
flagged=$(recsel -e "name = '"$pkg"'" -P flagged $apkpersist)
rss=$(recsel -e "name = '"$pkg"'" -P rss $apkpersist)
#If outdated display in red, otherwise green
if [ "$flagged" == "yes" ]; then
pver="$red[R]$pver$reset"
elif $(awk -v a="$pver" -v b="$rver" 'BEGIN{if (a <= b) exit 0; else exit 1}'); then
pver="$red$pver$reset"
else
pver="$green$pver$reset"
fi
#if the DB is missing an RSS feed, report yellow
if [ "$rss" == "" ]; then
pkg="$yellow$pkg$reset"
else
pkg="$pkg$reset"
fi
#if the rss version is 000, report yellow
if [ "$rver" == "000" ]; then
rver="$yellow$rver$reset"
fi
echo -e "$pkg $pver $rver"
done
else
for pkg in $(recsel -C -e "suppressed != '"true"'" -P name $apkpersist); do
pver=$(recsel -e "name = '"$pkg"'" -P version $apkpersist)
rver=$(recsel -e "name = '"$pkg"'" -P rssv $apkpersist)
flagged=$(recsel -e "name = '"$pkg"'" -P flagged $apkpersist)
rss=$(recsel -e "name = '"$pkg"'" -P rss $apkpersist)
if $(awk -v a="$pver" -v b="$rver" 'BEGIN{if (a < b) exit 0; else exit 1}'); then
pver="$red$pver$reset"
echo -e "$pkg $pver $rver"
elif [ "$flagged" == "yes" ]; then
pver="$red[R]$pver$reset"
echo -e "$pkg $pver $rver"
fi
done
fi
}
count_apks() {
count=$(recsel -C -P name $apkpersist | wc -l)
echo "Maintained: $count"
}
find_abandoned() {
apk_paths=$(rg -H -N "^# Maintainer:( )*$" $repo | sed 's/\n//g' | awk -F':#' '{print $1}')
for apk in $apk_paths; do
repo=$(echo $apk | grep -Eo 'testing.*|community.*|main.*' | awk -F'/APKBUILD' '{print $1}')
desc=$(grep -m1 pkgdesc $apk | awk -F'=' '{print $2}')
echo -e "$blue$repo$reset $desc"
done
}
find_maintainers_packages() {
#if we pass an email directly, simple search for it
if [ "$(echo "$1" | grep -o "@")" == "@" ]; then
maintainer="$1"
#Check to see if $1 is a file in aports, resolve the maintainer, and return all packages
else
file="$(rg --files $repo | rg --word-regexp "$1" | grep APKBUILD)"
for f in $file; do
if [ -f $f ]; then
maintainer="$(cat $f | grep "^# Maintainer:" | awk -F':.*<' '{print $2}' | sed 's/>//g')"
else
echo "$1 cannot be found, try searching for the maintainer's email or a different package."
exit 1
fi
done
fi
if [ "$maintainer" != "" ]; then
apk_paths=$(rg -H -N "^# Maintainer:.*<$maintainer>$" $repo | sed 's/\n//g' | awk -F':#' '{print $1}')
for apk in $apk_paths; do
repo=$(echo $apk | grep -Eo 'testing.*|community.*|main.*' | awk -F'/APKBUILD' '{print $1}')
desc=$(grep -m1 pkgdesc $apk | awk -F'=' '{print $2}')
echo -e "$blue$repo$reset $desc"
done
else
echo "Maintainer not found"
fi
}
#List the RSS address of all packages
list_rss() {
if [ "$1" == "urls" ]; then
for feed in $(recsel -C -P rss $apkpersist); do
if [ "$feed" != "" ]; then
echo $feed
fi
done
elif [ "$1" == "empty" ]; then
recsel -C -P name -e "rss = ''" $apkpersist
fi
}
#Reconcile DB with changes from git
rec_reconcile () {
if [ ! -f $apkpersist ]; then
cat > $apkpersist <<EOF
%rec: apk
%doc: APK Info
%key: name
%type: name string
%type: path string
%type: version string
%type: rssv string
%type: flagged string
%type: rss string
%type: suppressed string
%type: updated_on: date
%mandatory: name version
%allowed: name version rssv flagged rss updated_at
%auto: updated_on
EOF
else
cp $apkpersist /tmp/apks-$(date +%Y-%m-%d-%H%M%S).rec
fi
#Ensure we're not reconsiling on the wrong branch
cd $repo
if [ "$(git branch | grep '^\*' | sed 's/\* //')" != "master" ]; then
echo "Aports isn't on the master branch currently, please switch before reconciling."
exit 1
fi
#Grep the entire locally cloned aports repo for the defined maintainer, then create a csv file of import info for easier parsing.
#apk_paths=$(grep -H -r $email $repo | grep Maintainer | awk -F':#' '{print $1}') #this takes 2m
apk_paths=$(rg -H -N "^# Maintainer:.*<$email>" $repo | sed 's/\n//g' | awk -F':#' '{print $1}') #this takes 3s
for apk in $apk_paths; do
path=$(echo $apk)
apks=$(echo $path | awk -F'/APKBUILD' '{print $1}' | sed 's|'$repo'/||')
name=$(echo $apks | awk -F'/' '{print $2}')
pver=$(cat $path | grep "^pkgver=" | awk -F'=' '{print $2}')
#Attempt to resolve RSS path, if it doesn't exist in the db
crss=$(recsel -e "name = '"$name"'" -P rss $apkpersist)
if [ "$crss" == "" ]; then
src=$(cat $path | grep "^source=" | awk -F'=' '{print $2}' | grep -Eo "(http|https)://.*+" | tr -d \'\")
url=$(cat $path | grep "^url=" | awk -F'=' '{print $2}' | grep -Eo "(http|https)://.*+" | tr -d \'\")
#If the url ends in some trailing .extension, ie: .xz, .gz, .zip
if [ $(echo $url | grep -Eo "\.[a-z]+$") ]; then
#Try to make a valid RSS feed. This'll fail if the project name is different from package name (ie: ChezScheme is chez-scheme)
src_url=$(echo $src | sed 's|'$name'/.*|'$name'|')
rss=$(rss_feed $src_url)
else
rss=$(rss_feed $url)
fi
else
rss=$crss
fi
rdate=$(date)
if [ "$(recsel -e "name = '"$name"'" -P name $apkpersist)" == "$name" ]; then
recset -e "name = '"$name"'" -t apk -f path -s "$path" --verbose $apkpersist
recset -e "name = '"$name"'" -t apk -f version -s "$pver" --verbose $apkpersist
recset -e "name = '"$name"'" -t apk -f updated_on -s "$rdate" --verbose $apkpersist
else
recins -t apk -f name -v "$name" -f path -v "$path" -f version -v "$pver" -f rssv -v "0" -f flagged -v "no" -f rss -v "$rss" -f updated_on -v "$rdate" --verbose $apkpersist
fi
done
}
#Skim configured RSS feeds and repology api for outdated packages
check_outdated() {
outdated=$(outdated_apks | sed 's/:/-/g')
for pkg in $(recsel -C -P name $apkpersist); do
feed=$(recsel -e "name = '"$pkg"'" -P rss $apkpersist)
flagged=$(echo "$outdated" | grep -o "$pkg")
rssv=000
if [ "$feed" != "" ]; then
rssv=$(check_feed "$feed")
fi
echo "$pkg feed reports version $rssv"
recset -t apk -e "name = '"$pkg"'" -f rssv -s "$rssv" --verbose $apkpersist
if [ "$flagged" == "$pkg" ]; then
echo "$pkg has been flagged outdated on repology"
recset -t apk -e "name = '"$pkg"'" -f flagged -s "yes" --verbose $apkpersist
elif [ "$flagged" != "$pkg" ]; then
recset -t apk -e "name = '"$pkg"'" -f flagged -s "no" --verbose $apkpersist
fi
done
}
if [ "$1" == "-l" ]; then
list_apks $2
elif [ "$1" == "-lf" ]; then
if [ -z $2 ]; then
echo "list_rss requires either urls or empty as an argument"
exit 1
fi
list_rss $2
elif [ "$1" == "-cv" ]; then
check_outdated
elif [ "$1" == "-c" ]; then
count_apks
elif [ "$1" == "-fa" ]; then
find_abandoned
elif [ "$1" == "-fm" ]; then
if [ -z $2 ]; then
echo "find_maintainers requires either a package name or maintainer email"
exit 1
fi
find_maintainers_packages $2
elif [ "$1" == "-r" ]; then
rec_reconcile
elif [ "$1" == "-ur" ]; then
update_repo
elif [ "$1" == "-o" ]; then
outdated_apks
else
printf 'Usage: maintained.sh [-l] [-lf] [-c] [-cv] [-ur] [-r]
-l [all] | List packages & version. If red, out of date. If yellow, missing rss
-lf [urls|empty] | List package available or unconfigured rss feeds
-c | List total number of maintained apks
-fa | Find abandoned packages
-fm [email|package] | Find packages maintained by someone else
-cv | Check rss feeds & Repology for latest package version & cache
-r | Reconcile persistent recstore against aports
-ur | update aports fork master branch
'
exit 0
fi
Yes the script is a little janky, there's definitely a better way to do the python stuff, in fact the entire thing could probably be a python script, or nim, or go. Lets be honest, I really wanted a shell script. The difference in my mind is that this isn't a tool I'm writing to put into a repo to be consumed publicly. It is the horrible glue that keeps some thing together. That thing is my sanity maintaining so many packages. Shell glue is allowed to be ugly, as long as it's functional and reasonably easy to extend/maintain. I think this fits aptly! But it probably doesn't mean too terribly much to you if I don't explain what it's all for.
maintained.sh's single most important purpose in life is the generation of a rec file database that tracks information about all my packages. This allows me to do simple select style queries to determine if a package is out of date, or where it lives, or even just how many of these things I own the responsibility of. It is not proactive, but it is simple enough that I can work it into my daily workflow to constantly stay on top of!
The recfile that the script generates looks like lots of little blocks like this, simple plaintext.
name: sbcl
path: /home/durrendal/Development/aports/community/sbcl/APKBUILD
version: 2.3.9
rssv: 2.3.9
flagged: no
rss: https://github.com/sbcl/sbcl/releases.atom
updated_on: Mon Oct 9 09:37:37 EDT 2023
This is important for several reasons. 1) recutils is awesome, 2) it's plain text so if I outgrow it I sed/grep/awk it into insert queries and migrate to sqlite3, 3) did I mention recutils is awesome, that's because it's a whole bespoke plain text database system and if it didn't exist I'd probably try and do this using an org file. I can just quickly read this if I need it, I can parse it however I want. Perfect glue level functionality.
In essence this is just a love letter to the *nix design. Simple ubiquitous text interfaces everywhere, so that you can string together simple tools into custom solutions. I absolutely adore this, and apply it everywhere, from packaging to photography, it makes life so much more enjoyable.
Note:
I initially wrote this post in early August, during a fervor of productivity in the Alpine packaging realm. Then life got incredibly busy again, and I failed to ever publish this post even though it was essentially done save for spell checking. Sometimes "How I keep up with my packages" actually means I don't. As ingenuitive as I can be, I would not be able to manage any of this without the hard work and continuous assistance of other Alpine packagers, a sincere thank you goes out to anyone who's bumped anything I maintain!