(λ (x) (create x) '(knowledge))

The Infamous Droid

Revisiting My Mobile Workflow · April 24, 2022

My trusty 2012 Motorola Droid4, and a really cool fossil my son found!

Boy I feel like this post has been a long time in the making. I initially wrote about my mobile workflow back in May of 2020, but truthfully I had been using a Nokia N900 for a few months prior to that post. I had a very successful setup on PostmarketOS, but I didn't care for the distro, not the way I like Alpine. And the N900's hardware wasn't powerful enough for the development I was doing at the time, it may fair even worse today if I try to backpedal. The real limitation was ram, I don't mind waiting for a weak cpu to compile something, but if I simply run out of RAM trying to compile something in Golang or Common Lisp I will just straight up not have a good time.It's because of those limitations I started to look around for alternatives to the Nokia.

It's probably worth stating that I do like what PostmarketOS is doing, and the Nokia N900 hardware, no hate for either of these from me, it just wasn't what I needed.

Right anyways, the Droid. I had a friend in high school that had an original Droid, and frankly I always loved it. Just super cool looking tech. With the popularity of projects like PostmarketOS and Maemo Leste I really just guessed someone had at some point tried to find an alternate to the N900, an upgrade. I couldn't be the only one who felt 256MB of RAM was just too little to work with. Fortunately for me there was a little work being done on PostmarketOS that pointed to great success on the Maemo Leste project getting Linux on the xt894 Droid4. While PostmarketOS doesn't really have much support for that model, Maemo Leste loads just fine, has a custom kernel packaged, and a wiki full of usability tweaks for stabilizing the system. That's a huge boon when you start trying to run Linux on weird systems.

Really that's how I got here, standing on the shoulders of giants. tmlind has done most of the Linux kernel work to get the omap CPU, and other various bits of hardware working for the droid. He also maintains a droid4 specific kexec boot and alt alt, ones a custom bootloader capable of loading a zImage, and the other is a simple keyboard driver allowing you to correctly interface with the droid4 keyboard. It's stunning work honestly, and I wouldn't have gotten this thing working without all of his, and the omap Linux maintainers, hard work.

How I Use It

If you've read about my mobile workflow before, you won't be too surprised that it hasn't changed too much, mostly the beefier hardware has enabled me to add additional tooling, and better use the ones I was already using. I've broken these out into categories and talk generally about the what and the how of each.

Media/Entertainment

  • mocp
  • castero
  • mpv
  • epr/epy
  • newsboats
  • zangband
  • telnet elephant.org

Lets start with media, I think when most people think about the perks of a smart phone they quickly gravitate towards the immense amount of media you can instantly access. We've got podcasts, music, movies and TV shows, and so much more. I personally don't watch a ton of TV, but I do listen to audio books, music, and podcasts avidly. At one point I had a full X11 working on the droid, properly oriented, with my favorite color scheme and everything. I used i3 and my atentu config for that. However a secfix for X11 broke the patch that got all of that working. Once again this was something tmlind wrote, but it was rejected upstream and there's been no real progress getting it to work. Maemo Leste I think has their displays working correctly, so there's likely something I'm missing that I could patch and package, but I haven't gotten to it yet.

Getting back to the media thing, mocp is my golden goose. That's what I use to listen to audio books and music, I manually track where I'm at in my audio books, so if anyone knows a good cli audio book listener I'd love a suggestion. Castero is a little heavy resource wise, but is otherwise a rock solid podcast app, I used newsboat for a while, but didn't care for the experience. I still use newsboat to track the blogs I like and the git repos I maintain, it works exceptionally well. And epr/epy I cannot recommend more if you need a cli ebook reader! I occasionally read directly on my droid, but I find it's a lot easier to ssh in and read on my computer while i eat lunch. I've chewed through a number of books this way, and while it may be unconventional, I absolutely love it. Lastly, I do actually play some games on the droid, mostly small CLI games currently like Zangband and Elephant mud, but I enjoy the experience immensely. When I had X working I also played some GBA games on it, but not much.

Communication

  • neomutt
  • weechat

The next, or maybe this is the first, most used thing is probably communication. I have nowhere near the variety here because this tends to be the most focused out of anything I do. I use neomutt for email, and weechat for IRC. Boom, done. It's your typical suckless choice pretty much. It works incredibly well on low resource systems, it provides precisely the functionality I need, and it's flexible enough to work anywhere.

IRC is by far the primary use here, that lets me hit the Casa and chat with all of my friends, plus I get to play with all the cool bots we've built for our server lately. My contribution to that is a bot called Oscar he's named after Oscar the Grouch, and he exists to pull news headlines and post them into our #dumpsterfire channel. I wrote the little scamp because, well the current state of the world could be described aptly as a dumpsterfire, and anyone who consistently posts doom and gloom headlines is a bit of a grouch. That channel and bot exists because plenty of us are worried about the state of the world, and keeping a bead on it at least lets us know the world is still turning.

From a technical standpoint it's just a little Golang bot written on top of the go-bot library. Internally it just pulls and parses RSS feeds, stores the results in an sqlite3 db, and posts any new records to the channel. Dead simple. Freednode had a really robust newsboat in #news that inspired Oscar. Oh, and yes, I wrote, tested, and debugged Oscar on my droid!

Administration

  • lxc
  • ssh
  • drawterm
  • ansible
  • terraform
  • salt

At one point in my life I had to drag around a clunky multi pound Windows laptop with me everywhere I went. I was "on call 24/ 365" as my former employer put it. This was a major inconvenience, an absolute pain to put it lightly. I try and avoid maintaining Windows systems without sane FOSS tooling if at all possible, there's simply a better way manage 60 severs and 100 workstations than RDP and Bomgar. Microsoft has invested a ton of time in making powershell feel and operate like a what you'd expect a shell to work like. And while the syntax looks obtuse, it's honestly not that bad to script around solutions in. Fortunately as you can expect I can do a ton of this from my droid in a pinch. I wrote my powershell scripts in Emacs, I write my Ansible in Emacs, I write my salt states and terraformers and text documents and man pages all in Emacs. Simply put if I have Emacs I do write new administrative states as needed. If I'm out as a baseball game with my family and there's a literal crisis I can pop up a mobile hotspot and ssh into whatever. Sure you can manage this with termux too, I keep that on my android phone just in case too, but the lack of a physical keyboard there slows things down.

I administer my homelab this way too, if I'm traveling I just need my droid to jump in and work on whatever is broken. CI/CD crashed? lxc is on hand. Need to update a DNS record for my blog? Great terraform and git have me covered, I'll even make those changes persistent in version control. Wireguard lets me do all of this in a nice secure fashion from wherever I happen to be.


  Wait Will, did you just say you use wireguard on your droid? You literally just said during your last interview that your Linux kernel was broken and that couldn't work.

Oh yeah, that is right. When I spoke with dozens I couldn't use iptables or wireguard, the kernel was compiled without netfilter modules. Fortunately I was able to fix this shortly after that interview, nothing is more motivating than calling yourself publicly on a podcast, you bet I fixed it double fast afterwards! The kernel configuration tmlind is working on doesn't have these features, but I've added them to my droid4 bootstrap repo in case anyone is looking to get a droid of their own working (sorry it's a bit messy)! It's minimal configuration change, but I'm happy to have it working.

Development

  • git
  • glab
  • lxc
  • emacs
  • lynx
    • Languages
    • Common Lisp
    • Fennel
    • Golang
    • Lua

That segways very nicely into what is probably my most questioned workflow, development. When people find out about the droid they usually ask what I use it for, and then become a bit flabbergasted when I tell them I write lots and lots of code on it. I mean, if I'm comfortable doing admin work why wouldn't I also enjoy using it for programming? Oh, and package maintenance, though I'm pretty behind at the moment, I also primarily maintain my Alpine packages from the droid too!

When I do development work on my droid it's usually on one of the languages above, but I've also written some C, plenty of etlua for my blog (like this article!), and typically end up caching notes and reference material on whatever I'm learning using the droid. My workflow is what I would describe as offline first, but is largely enabled by tooling that I can only access over network.

What I mean by that is I maintain a rather large directory of offline notes on topics that span my entire professional career. Everything I've ever had to research, troubleshoot, create, or maintain has a collection of notes stashed in git oh the how, what, and why of it all. When a container crashes, I add more notes on making better LXC or docker containers. When I rebuild my NAS, I record all the steps leading up to the working systems deployment and then I log changes. This is really a force of habit, but having a tiny computer with a robust little keyboard I can jot down .org notes on while I work helps immensely. This has been years in the making, but keeping up with it has a massive benefit. If I ever find myself out and about with nothing to do, I can always work a bit on one of my projects with all the knowledge I've accumulated at my fingertips.

Since it's a neat data point, tree tells me this is a respectable collection of files.


  194 directories, 1321 files

The 2 core CPU and 1GB of RAM on the droid also enable me to actually compile on the system. I normally idle at about 40MB of RAM usage when I use the droid, so I have plenty of room to compile some of my Golang and lisp projects. Zram helps stretch that GB a little further too. Admittedly, I likely won't write anything crazy on it, but it's good for most of the small tooling I write

To date a number of the projects you can find on my Gitlab account have been written largely on the droid.

      tkts
    • Most of the initial code was written on my N900
    • The full rewrite from lua table to sqlite3 was done on my droid!
      sola
    • All of Sola has been written on the droid.
    • Most recently I started porting dozen's Chaote! STTRPG to Sola while on a flight from Maine to Florida.
      lambdacreate
    • Not strictly development, but I write most of my blog posts in some sense on the droid.

From an actual workflow perspective you'd probably imagine this feels a little cramped. I mean, it's a finger keyboard after all. I grew up using phones like this during high school though, so it feels pretty natural to crank out text on a two thumb keyboard. I'd even say I'm actually pretty quick at it, I don't ever feel the need to stare at the keyboard while typing, it just comes naturally. And the hardware has a nice clicky feedback with just enough distance between the buttons to prevent accidental button presses. Some people may not be comfortable with that, but I sure am. All of that is to say typing out a 2000-3000 word blog post is a breeze. Programming a little more tedious though. Symbols like {} [] and () are different levels in the virtual keyboard. I used a New Poker II mechanical keyboard as a daily driver at work, absolutely love that keyboard, and since it's a 3/4 setup I'm already very familiar with the leveled virtual keyboard paradigm.

With AltAlt running on the droid I can access symbols with a series of keystrokes. Pressing the SYM key twice will jump into AltAlt. So typing [ is Sym+Sym+CapsLock+9 on the droid and { is Sym+Sym+Tab+9. If I want parenthesis I just need to do Capslock+9. To explain that more clearly Sym = Alt, Capslock = Shift. Alt+Alt+Shift+9 = [, so on and so forth. Writing lisp/scheme is thus a breeze on the droid since parenthesis are just a shift away. Writing Fennel, Golang, C, and other languages that use brackets are less fun, but what's two button clicks between friends? It certainly doesn't disrupt my workflow or prevent my productivity.

When I'm testing code I do usually compile and run it on the droid, so most of my software should work on low powered arm systems on purpose. There's noticeable delay building large Golang projects but otherwise it works well. I've even cross compiled to windows x64 from the droid in a pinch, it works very well if you're patient.

The size of the screen is obviously kind of small, when I had X11 working scaling the font up a little (I think I used 16 or 20pt font) worked great for most tools. And I just had an hotkey in i3 to launch terminals with bigger or smaller fonts as necessary. Without X11 the screen can be hard to read in direct sunlight unless you crank the back light up, in low light it's super legible even with small font, and the back light keyboard is excellent for late night flights when you want to crank out a blog post about your weird Linux phone without disturbing your neighbor. Regardless of X11 or no X I make heavy use of workspace and tmux panes to efficiently work with references and applications.

Most Recently

Most recently I had to travel for work, I usually bring a personal laptop and my work laptop with me, but this time I forewent the personal laptop. Mostly because I had to bring 2 firewalls and a mini PXE server with me to the remote site. As you can imagine I got plenty of strange looks from the TSA.

Anyways, these types of trips are great ways to uncover bugs and put my system to the test. Most of the trip I was focused on imaging systems and didn't really use the droid during the day. When I did use it it was just to access my knowledge repository while troubleshooting issues with Windows. It was nice to have, but I could have also accessed this on the Gitlab website so it was only a minor boon. However I made heavy use of the droid during my flights in and out. I neglected to bring along any form of entertainment with me, and the bulk of my travel was late night well after working hours. Keeping myself awake and entertained was my primary focus.

To do so I spent most of the time listening to podcasts with castero or music with moc while copying the a STTRPG ruleset from a pdf on my phone into a sola ruleset. Those rulesets are all written as Lua tables, and the sola application is written in Fennel. I probably pressed the sym key a way too much, but on the first leg of my flight in I was able to get the bulk of the rules created and begin testing the sola handler functions. I had hoped I'd get enough done to play the STTRPG on my flight back home, but I wasn't that efficient. I could have ported the full ruleset much more rapidly if I had my netbook with me. I can definitely tell I traded a bit of efficiency/speed for portability here.

In the terminal while I waited for my connecting flight I hopped on the airport wifi and threw together a new set of iptables rules for the droid. It feels good to put a real firewall on this device finally. My ruleset unfortunately was a bit aggressive and blocked my wireguard VPN, so I ended up disabling it and then found that I can't reach my wireguard VPN at all, despite it working before I left. I don't do anything special with my wireguard currently, but I might need to setup DNAT rules in iptables and set up forwarding on the droid to get it to punch through the airport firewalls and reach my remote. I wasn't able to actually test that theory during my trip, but I've done a very similar config with systems behind CGNAT LTE networks and it works very well once configured. Weechat though worked flawlessly and it was nice to chat with the casa while I was waiting.

The rest of the trip was dead air on the droid, as soon as I touched down it was go, go, go; and I only got one cup of coffee the entire time. Unlike my friend dozens I'm an avid coffee drinker, and while I function perfectly fine without it, I don't forgo it unless I simply must. If I couldn't get coffee, well, you can probably guess I also couldn't play with my droid.

On the way back home I listened to the latest episode of Tildewhirl, it was great learning about Piusbird. The airport I was flying out of unfortunately blocks SSH and IRC, so I was restricted to offline work. Still after hearing about Piusbird's teletype project I felt inspired to program and worked a bit more on sola, and mulled over the idea of doing a teletype project just to learn how to make one. I always liked the ones in the TV station, mostly because I knew they were little embedded Linux systems, and I've been dabbling in TUI programming in Golang a lot lately, primarily so I can make a BBS for the casa and it seems like a good TUI style project to add onto the books and practice my skills.

That brings us up to date now, where I sit pensively typing out the last of this blog post from a flight back to Maine. I'd say things went as well as expected. My pain points above would have been similar on a laptop since they're related to bypassing firewalls and configuring network equipment and not really restrictions caused by the droid itself. I will say that captive login portals are a massive pain in the ass to deal with though. I've had some success with lynx, but it's very hit or miss. Most of the time I ended up using my phones hotspot to deal with captive portals because I just couldn't get to them on the droid. When X11 was working I just used Firefox and everything was fine. I'll have to push fixing X higher on my todo now that I have WG support on the droid.

Closing

You can probably tell that I consider my droid a stalwart companion by now. I travel with it everywhere I go, because it's to date the only truly portable device that fits most of my use cases. It's also an easily identifiable device, nobody at the TSA is going to bother you for having a cell phone, there's never an arm room issue using a tiny 5in mobile to write code on a plane. And at the end of the day, you can always find room for a couple of ounces for something so useful, at least I always do.

Anyways that's my workflow for this little guy, I pack as much functionality into a low powered device as is humanly possible! If you're skeptical about it, why don't you check out my presentation from Fennel Conf 2021, I gave my entire presentation on my droid while SSH'd into it from a T470s Thinkpad (unfortunately I have no found a good way to use it for Jitsi).

Also after listening to a few episodes of dozen's new podcast I want to steal one of his ideas because it's really cool. He does this "letters to the editor" type segment where people can write in to him and ask questions or ask for clarifications. I'll add something more official to the site when I get a chance, but I want to do something similar!

If you have any questions on anything I've written about, or would like me to talk about a certain topic I've got experience on, or that you think I may have an interesting opinion about please reach out to me. I would love to hear your thoughts and suggestions. Feel free to reach out to me via IRC or email if you have anything to suggest!

Post Trip

Right so I STILL haven't posted this blog post, totally unsurprising. Over the Easter break I actually made time to address the issues I brought up in the post above. Happy to say all of them were entirely resolvable.

Wireguard was working correctly on the droid, however I stupidly configured my Mikrotik to only allow connections from my own WAN. I checked the wireguard service before I left, it worked, then it just didn't when I left. Double check your configs and don't make production changes last minute.

After fixing that I realized I forgot to add fuse support to my new wireguard enabled kernel, so even if wireguard worked sshfs would not have, which means stream music off my NAS would have been less streamlined and unpleasant. Obviously that got fixed with a rebuilt, and I added 9p2000 support because I could.

And probably the best update, I fixed my xorg-server! Turns out I really just needed to spend a little time debugging the compilation, all that changed was a variable call, tmlind's patch was still very much valid! Additionally I needed to add my user to input & video and install libinput due to changes in the way Alpine handles those permissions, but those were expected changes. The photo at the top is post trip fix xorg-server Droid!

Bio

(defparameter *Will_Sinatra* '((Age . 31) (Occupation . DevOps Engineer) (FOSS-Dev . true) (Locale . Maine) (Languages . ("Lisp" "Fennel" "Lua" "Go" "Nim")) (Certs . ("LFCS"))))

"Very little indeed is needed to live a happy life." - Aurelius