Back At It Again - OCC Day 1

Alpine Linux & an Acer ZG5 · July 7th, 2023

It's been a hot minute since I updated the old blog. Not for any lack of trying, I have half finished posts all over the place, lots of them waiting for me to finish developing film/scan developed negatives. Sadly, I don't have a fix for that at the moment, because we've just started the OCC for the third year in a row! And because of that most joyous occasion I'll be unable to scan any of my negatives, the software to run the scanner is just too heavy for my poor netbook. Oh well, we'll see what happens after this years challenge.

Anywhos, OCC. This year I'm rocking my Acer Aspire One ZG5, it's the smaller version of the D255 I threw 9front on last year. This puppy has a 2 core Atom n270, 1gb of ram, and a 140gb 5400rpm ATA drive, honestly pretty solid specs for a laptop from 2007ish. And unlike last year I'm not using some weird ancient OS for a week, I'm using the tried and true Alpine Linux.

You see this year I have lofty goals. With the online time limitations imposed last year, and the fact that I was traveling for work during the challenge, I barely feel like I got to participate at all. Sure I got a cool 9front install before the trip, but my usage was minimal. And we're imposing no such limitations on this years challenge. All of that leads me to believe that the intent and purpose is to do as much of my regular work inside of the minimal specs during the challenge week! We're going to be productive and do as much as possible with as little as possible. And because I just don't want to wait, we're starting today instead of on the 10th, though I'll probably still end on the 17th.

I actually spent the last couple of weeks preparing for all of this. I was using my ZG5 to play with Sway, and it just runs poorly on these low specs, so I did a complete wipe and went with ratpoison as my WM, just to change things up. I could have used i3, but then it would literally just be a big version of my droid. And while I did liberally copy over my weechat, emacs, ssh, moc, and lxd configurations, I really didn't want this to be a week of me "using my droid but not really using my droid". At least with a change of WM I get to branch out and try something new.

I almost forgot to include a photo, here's my ratpoison setup in all it's glory. This screenshot is timestamped 06/09/23, only got way too excited about this a whole month too early.

Day One

If it isn't obvious already, I'm writing this in retrospect. Thus far the entire evening has been spent listening to music using moc (chewing through Flor's albums specifically), chatting with friends on weechat, and working in emacs, lxd, and with verkos to create installation scripts for a Grafana, Loki, Nginx server with a promtail client script. All of that has been swimmingly easy, from compiling verkos using nim, to living inside of emacs, it feels exactly like home. Most of my daily workflow is command line based already, all of my dev work happens inside of emacs, and as long as I can spin up LXC containers for testing I more or less have a one to one re-implementation of my typical development environment.

It turns out that I kind of accidentally adopted light weight and optimized low resource workflows due to my proclivity towards making that droid4 a daily driver. And I guess from my perspective, CLI tooling is more eloquent then big clunky graphical IDEs or applications.

But none of that prevents me from needing to work on those sorts of things. See the whole Grafana/Loki thing is me hunting for a sustainable, salable SIEM solution that I can deploy in my homelab, and possibly at $work if it fits the bill. While I feel like I do a decent job monitoring for service level issues using Zabbix, I don't think I have enough visibility into the logging information throughout the systems I manage. Fortunately for me, it turns out that Loki is dead simple to configure and deploy, Promtail is just as easy to point at various log files, and even Grafana is easy to work with, to a point. Trying to configure Grafana dashboards using Firefox on my Acer was a lesson in patience like no other. The clunky javascript heavy frontend was simply not designed to run on this old hardware.

But I'm not hating on Grafana, I actually really like what it brings to the table from a data visualization perspective. I'm just using the wrong tool right now. When the challenge is over I'll tinker more with it. Tonight was spent entirely on those Verkos scripts to deploy & configure Loki & Promtail for testing, and all of that was a dream.

Maybe it's entirely unnecessary, especially since you can look at Verkos on Gitlab but I really love how these shell scripts turn out. It's just a big amalgamation of previously written shell snippets, but gluing them all together using a little bit of yaml, and letting Verkos generate the script for me is a massive reduction in mental capacity. I really find that I hone in on doing the thing, and don't get caught up so much on the scripting itself. I just add features piecemeal until I get a working thing. Here's what I ended up with for the Loki server, pretty nifty if you ask me, especially since you just need a plain alpine linux install and to copy over this script and run it to get a working Loki stack!

#!/bin/ash
set -ex

#Usage: repos
repos() {
	cat > /etc/apk/repositories <<EOF
http://dl-cdn.alpinelinux.org/alpine/$1/main
http://dl-cdn.alpinelinux.org/alpine/$1/community
http://dl-cdn.alpinelinux.org/alpine/$1/testing
EOF

	apk -U upgrade
}

#Usage: pkgs 'htop tmux emacs' [update]
pkgs() {
    if ["$2" == "update"]; then
	   apk update
	fi

	if ["$1" != ""]; then
		apk add $1
	fi
}

#Usage: crontab_base [blank]
crontab_base() {
	if ["$2" == "blank"]; then
		cat > /tmp/new.cron >>EOF
		EOF
	else
		cat > /tmp/new.cron <<EOF
# do daily/weekly/monthly maintenance
# min	hour	day	month	weekday	command
*/15	*	*	*	*	run-parts /etc/periodic/15min
0	*	*	*	*	run-parts /etc/periodic/hourly
0	2	*	*	*	run-parts /etc/periodic/daily
0	3	*	*	6	run-parts /etc/periodic/weekly
0	5	1	*	*	run-parts /etc/periodic/monthly
EOF
	fi
}

#Usage: crontab_append '*/15 * * * * /usr/local/bin/atentu -m > /etc/motd'
crontab_append() {
	printf "$1\n" | tee -a /tmp/new.cron
}

#Usage: apply_crontab
apply_crontab() {
	crontab /tmp/new.cron
}

#Usage: loki_conf
#Variables:
loki_conf() {
	if [-f /etc/loki/loki-local-config.yaml]; then
		mv /etc/loki/loki-local-config.yaml /etc/loki/loki-local-config.yaml.bak
	fi

	if ["$(grep loki /etc/shadow)" == ""]; then
		useradd -M -U -r loki
	fi

	cat > /etc/loki/loki-local-config.yaml <<EOF
auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
ingester:
  wal:
    enabled: true
    dir: /tmp/wal
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h
  max_chunk_age: 1h
  chunk_target_size: 1048576
  chunk_retain_period: 30s
  max_transfer_retries: 0
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /tmp/loki/chunks
compactor:
  working_directory: /tmp/loki/boltdb-shipper-compactor
  shared_store: filesystem
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
chunk_store_config:
  max_look_back_period: 0s
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
ruler:
  storage:
    type: local
    local:
      directory: /tmp/loki/rules
  rule_path: /tmp/loki/rules-temp
  alertmanager_url: http://127.0.0.1:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true
EOF
}

#Usage: nginx_proxy host_port proxy_addr
#nginx_proxy 80 127.0.0.1:3000
#Variables: ssl_crt ssl_key
nginx_proxy() {
	if [! -d /etc/nginx/sites-available]; then
		mkdir /etc/nginx/sites-available
	fi
	
	cat > /etc/nginx/sites-available/grafana.conf <<EOF
map \$http_upgrade \$connection_upgrade {
	default upgrade;
	'' close;
}

upstream grafana {
		 server $2;
}

EOF

	if ["$1" == "80"]; then
		cat >> /etc/nginx/sites-available/grafana.conf <<EOF
server {
	   listen 80;

	   location / {
	   			proxy_set_header Host \$http_host;
				proxy_pass http://grafana;
	   }

	   location /api/live/ {
	   			proxy_http_version 1.1;
				proxy_set_header Upgrade \$http_upgrade;
				proxy_set_header Connection \$connection_upgrade;
				proxy_set_header Host \$http_host;
				proxy_pass http://grafana;
	   }
}
EOF

	elif ["$1" == "443"]; then
		cat >> /etc/nginx/sites-available/grafana.conf <<EOF
server {
	   listen 443 ssl;
	   ssl_certificate $ssl_crt;
	   ssl_certificate_key $ssl_key;

	   location / {
	   			proxy_set_header Host \$http_host;
				proxy_pass https://grafana;
	   }

	   location /api/live/ {
	   			proxy_http_version 1.1;
				proxy_set_header Upgrade \$http_upgrade;
				proxy_set_header Connection \$connection_upgrade;
				proxy_set_header Host \$http_host;
				proxy_pass https://grafana;
	   }
}
EOF
		fi
}

#Usage: nginx_activate name
#Variables:
nginx_activate() {
	ln -s /etc/nginx/sites-available/$1.conf /etc/nginx/http.d/$1.conf
}

#Usage: enable_services default 'lighttpd rsyslog samba iptables'
enable_services() {
	for service in $2; do
		rc-update add $service $1
	done
}

#Usage: reboot_system
#Variables
reboot_system() {
	reboot
}

repos edge
pkgs "loki loki-logcli loki-promtail grafana iptables syslog-ng nginx shadow"
crontab_base
crontab_append "0 2 * * 5 /sbin/apk -U -a upgrade"
apply_crontab
loki_conf
nginx_proxy 80 127.0.0.1:3000
nginx_activate grafana
enable_services boot "syslog-ng"
enable_services default "crond iptables loki nginx grafana"
reboot_system

Anyways, enough of today, lets talk about the rest of the OCC. I have a few goals I want to accomplish.

Goals

• Push the envelope. If there's something I usually do I want to try and do it with this ancient junk.
• Infrastructure improvements. I want to use this slightly reduced distraction period to implement a NIDS, HIDS, & SIEM in my homelab, at very least.
• Digital Photoghraphy. My son thrifted a Sony DSC-S75 Digicam recently, and I'm helping him refurbish it. Assuming that goes well I want to run it through some thorough tests during the OCC and replace my film cameras with it for the duration of the challenge. That should include editing the photos, because I THINK this thing can handle gimp.

Nothing too crazy, but I think I have a few things to focus on, and hopefully lots to write about. I would say podcast about too, but the entire point of LTRG is to do the podcast on the droid, so I'll probably hold off on that. Maybe I'll reconsider that point, it would be fun to do a field day with the netbook if I can get somewhere interesting to do it. Maybe a day trip to Monhegan is in the future..