(λ (x) (create x) '(knowledge))

All new Lambdacreate

Same great taste! ยท July 13th, 2024

You may or may not have noticed that things around here seem different. From my perspective they feel really different! But except for some really amazing CSS changes, it's all fairly opaque to you dear reader. That's because I've spent what little free time I've had working on the bits and pieces that make lambdacreate "tick". You could image that the innards of this little blog are like a beautiful piece of clockwork, with bespoke pieces of beautiful code enmeshed into a tight and highly organized system.

Of course, that would be wrong, this blog is more like my junk drawer. It's filled with half thought ideas, learning on the fly, and sometimes really bad ideas that should never have made it to production. Well, that's how it looked until recently at least. I've refactored most of Lapis backend, making sweeping changes to the way that I handle access and maintenance of the database components, render posts, and even the infrastructure that lambdacreate itself rides on. It's a great bunch of changes, that feels very literally like I've cleaned out my junk drawer.

Of course, in the process of cleaning out ones junk drawers, you usually find old memories and interesting bits and bobs you had long forgotten about. This post, is precisely that. Just a quick reflection on some neat things I learned while cleaning up. At the end of the day, this is the same great Lambdacreate, just with some improved maturity.

Migrations in Lapis

It turns out, that Lapis natively supports using sqlite3 databases now! This was absolutely not the case when I originally set out to create Lambdacreate. I remember specifically using a janky set of lua tables instead of an SQL database as my meta data backend because the idea of managing a MYSQL or PSQL instance for my blog just sounded unfun.

Now don't get me wrong, SQL is a need to know skill-set in my field of work, in hindsight if I really wanted/needed to learn SQL quick and wanted to really hone in on a specific flavor of it I'd probably choose PSQL and just accept the complexity and weight costs.

Obviously this janky lua table idea only worked for so long though, because I used lua-lsqlite3 to add some relatively unsafe sql handling to the site and just accepted it for the longest time. All of this came to a head when I started working on Katalogo with Mio.

Using the momentum and changes from that project, I reworked the DB handling for Lambdacreate to not only use the Lapis Migrations system, but also to use the much much safer sql querying methods that are internal to Lapis. The net result is that it's magnitudes of order easier to create new databases, to grok what the database format for the site is, and queries just work so much more reliably.

If you're working with a framework, use the tooling built into it if you can , there's usually a reason for it to exist.

Kind of like Markdown

You could write out html, and try to figure out the formatting and all the fancy highlighting. You could kid yourself into believing that you will some day embed lua scripts into a blog post to make it interactive. You may take this delusion straight down to design paradigm and thus use Etlua templating to write all of your blog posts for several years when there is a perfectly reasonably, well used, and easy to use framework agnostic solution literally in front of you. Namely markdown.

All of the blog posts and podcast show notes are now markdown files, and while that adds an additional dependency to the Lapis stack and another package to my burgeoning Alpine contribution, it is so so so worth it to just be able to write without stupid html tags getting in my way!

But wait, that must have meant converting all of those etlua files right? Oh absolutely, I could have hand converted them if I needed to, but that sounds like literal hell. I'm lazy. Not even for a person project could I be bothered to hand convert files, that's such a dull task.

But you know what isn't dull? Finding yet another excuse to learn more about Ruby! I'm on a roll here lately, everything has been Ruby themed, and I have to say, I wasn't certain I'd like it when I first picked it up, but it's growing on me.

Here's a little script I threw together to convert all of the etlua files into markdown, it was honestly really straight forward to throw together.

#!/usr/bin/ruby
# Convert Etlua files into Markdown
require 'reverse_markdown'
require 'optparse'

# Given an etlua file, determine the name, convert it to markdown, then render it to a new file with the same name and a .md extension.
def parseFile(path)
  # Resolve path name & file name
  dir = File.dirname(path)
  name = File.basename(path, ".etlua")
  # Read in the contents of the etlua file
  etlua = File.read(path)
  # Convert it to markdown
  md = ReverseMarkdown.convert(etlua)

  # Create a new file name name.md in the same directory as the etlua file, and populate it with the markdown conversion
  File.open(dir + "/" + name + ".md", "w") { |file| file.write(md) }
  puts path + " converted to markdown as " + name + ".md"
end

# Iterate over a directory invoking parseFile on any found .etlua files
def parseDir(path)
  Dir.glob(path + "/*.etlua") do |etlua|
    parseFile(etlua)
  end
end

# etlua-to-md.rb -f views/posts/1.etlua
# etlua-to-md.rb -d views/posts
def main()
  # Setup option handing
  options = {}
  OptionParser.new do |opt|
    opt.on('-f FILE', "Path to an etlua file") { |o| options[:file] = o }
    opt.on('-d DIR', "Path to a directory of etlua files") { |o| options[:dir] = o }
  end.parse!

  # If you don't pass any args, throw an error.
  if options.length < 1 then
    puts "The path to an etlua file, or a directory of lua files is required."
    exit
  end

  # Configured reverse_markdown so that it knows how to convert <pre><code> blocks
  ReverseMarkdown.config do |config|
    config.unknown_tags     = :bypass
    config.github_flavored  = true
    config.tag_border  = ''
  end

  # If given -d path and not -f path, iterate over a directory
  if options[:file].nil? and !options[:dir].nil? then
    parseDir(options[:dir])
  # If given -f path and not -d path, convert a single file
  elsif !options[:file].nil? and options[:dir].nil? then
    parseFile(options[:file])
  else
    # Anything else flags an error.
    puts "Incorrect arguments provided."
    exit
  end
end

main

The one thing I will say about Markdown, is that I can never remember how to handle links. So if someone notices something isn't linked properly feel free to contact me and let me know, I try to double check, but it doesn't always happen.

Migrating Data from one SQLite DB to Another.

This was a neat trick I figured out on the fly, due to the changes I made to the way the database is provisioned, but the schema remained relatively static. And because I wanted to keep several portions of the database in tact, I found a relatively easy way to partially import portions of the existing database to the new system.

What I'm talking about is attaching the old db, and them using plain old SQL queries to import data from one to the other.

Just jump into the old database.

sqlite3 lc..db-old

And then attach the new database and insert into it the output of select everything from the old database!

ATTACH DATABASE "lc.db" as 'new';

INSERT INTO new.shows SELECT * FROM shows;
INSERT INTO new.episodes SELECT * FROM episodes;
INSERT INTO new.projects SELECT * FROM projects;
INSERT INTO new.posts SELECT * FROM posts;

Neat right? All I really did was drop the old authentication information table, which could maybe have been handled a different way, but this felt really cool to me honestly.

Which brings me to authentication

It turns out that trying to roll your own authentication paradigm is bad. This should be surprising to literally nobody, myself included. But that's pretty much what I was doing until recently. See the paste service on my site needs some level of authentication, I don't want it to be used as a public paste-bin, it's my own personal digital junk drawer.

So previously, the authentication mechanism was attempting to implement a hand-rolled version of PBKDF2 auth, without actually referencing the spec for it, or even understanding what it was or that it existed at all. I just thought the idea of encrypting my hashes was good. And it is, but what I did was not.

Initially things started out pretty okay. We had a randomly generated string, which we base64 encoded and then sha256 hashed to use as the signature for our encryption. We would salt and hash each credential so you'd need the signature and the hash to make an auth.

function auth.encode(secret, str)
   --Encode a string in base64, then hmac sha256 it returning msg.signature
   local msg = mime.b64(str)
   local signature = mime.b64(auth.hmac256(secret, msg))
   return msg .. "." .. signature
end

function auth.decode(secret, msg_and_sig)
   --Attempt to decode a msg.signature pair.
   local sep = "%."
   local msg, sig = msg_and_sig:match("^(.*)" .. tostring(sep) .. "(.*)$")
   if not (msg) then
         print(inspect({"Invalid format"}))
         return nil, "Invalid format"
   end
   local sig = mime.unb64(sig)
   if not (sig == auth.hmac256(secret, msg)) then
         print(inspect({"Invalid signature"}))
         return nil, "invalid signature"
   end
   return mime.unb64(msg)
end

--Encoding example
--local secret = "password"
--local hmac = auth.encode("This is a hidden msg", secret)
--print(hmac) => VGhpcyBpcyBhIGhpZGRlbiBtc2c=.y0poUMhGvi9F8B6Gd4xZPJpbqpDhM6xYP/ySeF0lTNU=
--print(auth.decode(hmac, secret)) => This is a hidden msg

Well that's fine in practice, but in implementation, the result that ended up going into production was less than ideal. This quasi combination of bcrypt and this weird home made salt/hash thing is what was the underlying authentication system for a while. And while it did secure the paste bin, by virtue of require some random string be provided to the API endpoint for it to work, it had some nasty pitfalls.

	  --Decrypt cached credential & signature using salt
	  local dec = auth.decode(salt, info[1])
	  if dec == nil then
		 return falses
	  end

	  --Hash decrypted credential
	  local cred_digest = bcrypt.digest(dec, '12')

	  local cred = mime.unb64(key)
	  --Compare provided credential against generated hash
	  local cred_verify = bcrypt.verify(cred, cred_digest)

First, and most problematic, more or less this is just a very convoluted and pointless way of having obscure password protection. The credential gets stashed in the DB in reversible encryption instead of properly hashed. Terrible terrible design on my part. Secondly due to the really shitty salt/hashing the random credential could only be of certain lengths and couldn't contain specific characters.

Ever signed up somewhere and tried to use your password manager only to be told your password couldn't container $&^%@ etc? Yeah, I totally made one of those.

I've hated this thing pretty much since I made it. It was good enough for my one user tooling, but there's always room for improvement, and if I was going to overhaul a bunch of stuff this absolutely was getting thrown on the operating table and dissected.

In the end, I did a lot of reading, and went with the tried and true bcrypt. It isn't perfect, but the API currently doesn't do a whole lot either. Down the road as the site continues to grow in functionality I'll migrate to PBKDF2.

And finally, these nice new digs

It's subtle in some places, but definitely noticeable on mobile, especially if you're looking at the images. Lambdacreate has a great new look thanks to several change suggestions Mio made recently. They not only provided examples, but went so far as to mock the site, and explain the changes in inline comments.

I'm sincerely grateful for this, I don't personally think I'm the best at design, and CSS has always just been opaque hoodoo to me. Literally the entirety of my CSS file was cribbed from random places across the internet and pasted together like some kind of Frankenstein's monster. I've honestly always been frustrated and ashamed by that fact, but it really didn't click for me until recently.

Thanks Mio! For making LC look so much nicer, and teaching me something in the process!

End

Phew, I started writing this back at the end of June and it took me maybe 3 weeks to finally finish it. Kind of ridiculous, but such is life right now. Fortunately the OCC is on the horizon, so we'll be back here shortly!