(λ (x) (create x) '(knowledge))

Lisp, Make, & Esper

Homogenous Packaging for Heterogenous Environments · October 7th, 2020

At Chenmark a lot of what I've been doing up to this point has been building infrastructure, but every once in a while I get to step back and write a tool with a purpose to bridge a gap. That's a whole thing unto itself in the small business realm. A lot of software in production could be considered legacy, and has so entrenched itself in the day to day operation of the business, that the thought of upgrading it gives people heart palpitations. When your entire factory production line relies on the information you get from that legacy system, can you blame them? But that's hardly a way forward.

Technology is always changing and we cannot pigeon hole ourselves into what we consider safe and sound. We also can't run stable businesses on bleeding edge systems. Try and run an Arch Linux server in production, the guys at Jupiter Broadcasting make it seem funny, but it's more effort than it's worth. That's why we do a lot of gap bridging. Getting people off of legacy systems means gently guiding them towards more optimized modes of operating, by taking the legacy data out of the legacy systems, and upgrading the interfaces they use to access it.

Great! We throw together some web app on top of a legacy database right? Sounds like a Frankenstein misery. Rather we take the legacy (and often times proprietary) databases, and write our own integrations. Migrate the data out and import it into a modern database, like Postgres, mock the interface and slowly build optimization algorithms around their data, and usage. That process however is very rarely a cut and dry, move everything from one place to the other ordeal. It's usually a lot of hands on user training, and consistent ex-filtration of data as the users continue to rely on their familiar legacy systems.

Thus I've been learning a lot more about deploying software on Windows systems, to create system services to ingest, manipulate, and ex-filtrate data. And being myself, I really didn't want to learn how to do things a new way when GNU packages make for Windows, SBCL 2.0.0 is a quick download away, and Quicklisp is itself a self contained lisp program. To build and deploy a Common Lisp service to handle that, I really don't need much! I'm happy to say because of this, building the application boiled down to a very simple change to my typical Linux Common Lisp makefile.


LISP ?= sbcl
LISP_FLAGS ?= --no-userinit --non-interactive
HOST ?= "linux"
QUICKLISP_URL = https://beta.quicklisp.org/quicklisp.lisp
QUICKLISP_DIR = quicklisp

all:
    $(MAKE) quicklisp
    $(MAKE) omni
    $(MAKE) clean

ifeq ($(HOST), "linux")
quicklisp:
    mkdir -p $(QUICKLISP_DIR) ;\
    curl --output quicklisp.lisp $(QUICKLISP_URL)
    $(LISP) $(LISP_FLAGS) \
                --eval '(require "asdf")' \
                --load quicklisp.lisp \
                --eval '(quicklisp-quickstart:install :path "$(QUICKLISP_DIR)/")' \
                --eval '(uiop:quit)' \

    $(LISP) $(LISP_FLAGS) \
                --load ./quicklisp/setup.lisp \
                --eval '(require "asdf")' \
                --eval '(ql:update-dist "quicklisp" :prompt nil)' \
                --eval '(ql:quickload (list "asdf" "uiop" "dexador" "modest-config" "unix-opts"))' \
                --eval '(uiop:quit)'
endif

ifeq ($(HOST), "windows")
quicklisp:
    mkdir $(QUICKLISP_DIR)
    powershell -command Invoke-WebRequest -Uri $(QUICKLISP_URL) -Outfile quicklisp.lisp
    $(LISP) $(LISP_FLAGS) \
                --eval '(require "asdf")' \
                --load quicklisp.lisp \
                --eval '(quicklisp-quickstart:install :path "$(QUICKLISP_DIR)/")' \
                --eval '(uiop:quit)' \

    $(LISP) $(LISP_FLAGS) \
                --load ./quicklisp/setup.lisp \
                --eval '(require "asdf")' \
                --eval '(ql:update-dist "quicklisp" :prompt nil)' \
                --eval '(ql:quickload (list "asdf" "uiop" "dexador" "modest-config" "unix-opts"))' \
                --eval '(uiop:quit)'
endif

omni:
    $(LISP) $(LISP_FLAGS) --eval '(require "asdf")' \
        --load ./src/omni.asd \
        --load ./quicklisp/setup.lisp \
        --eval '(ql:quickload :omni)' \
        --eval '(ql:quickload (list "uiop" "dexador" "modest-config" "unix-opts"))' \
        --eval '(asdf:make :omni)' \
        --eval '(quit)'

ifeq ($(HOST), "linux")
clean:
    rm quicklisp.lisp ;\
    rm -rf $(QUICKLISP_DIR)
endif

ifeq ($(HOST), "windows")
clean:
    rem quicklisp.lisp
    rmdir -r $(QUICKLISP_DIR)
endif
 

Yes it's a little bit obtuse, I've obviously defined the same thing twice with very modest changes, a curl call turns into a powershell cmdlet, we change the syntax of our rm calls. It really isn't much at all, but should should just speak to the robustness of the system itself. The same make configuration works 98% of the way between Linux and Windows. Due to the design of Common Lisp's package manager, it can just be bootstrapped into the lisp image of our choosing. The only thing that really changes between Linux and Windows is how we fetch that part of our build system.

Maybe that little bit of information can help someone who's just getting started, or like me, finds themselves suddenly thrust into deploying software for an OS they detest. At the end of the day, it doesn't really matter what your personal opinions are, adapt your methodologies to reach as many people as is humanly possible, it's well worth it.

At this point though I've compiled everything from C, to Go, to Common Lisp, and a dozen different things in between. A lot of the time I don't know anything more about the language than how to use make, and skim through the source for context. Make is that universal equalizer that makes distribution happen. But it isn't really a package manager, it's just the build. And while I can (and often do!) package anything I want or find interesting for Alpine Linux, I can't effect the Debian, or RHEL packaging systems, not for a lack of desire, but from time. That limits my ability to share things with friends, to package applications in containers, and even to some point distribute them in an ad hoc fashion.

Esper

Let's talk about Esper.

Esper is a build system written in Fennel, that can be compiled down to a single binary, or a stand alone Lua script requiring only Lua itself. It's a meager 200 lines of Fennel which provides the following functionalities.

  • Source Fetching
    • From Git, URL, or Locally
    • Extraction of tarballs and zip files
  • Dependency Installation
    • For Alpine, Debian, and RHEL based systems
  • Source Building
  • Source Installation
    • With permission & renaming functions
    • for both files and directories

Okay, so maybe that feature list isn't as impressive as a full featured package manager, but it's a work in progress under active development. And it exists to fit a specific niche. Systems like luarocks and quicklisp are so helpful because they are ported everywhere the language they support can be found, but that doesn't exist for package managers. Yes you have make, and you can just add "install these packages on Debian systems" and "these on alpine" calls to your makefile, but that's not really the point of make. Esper is my attempt to make a more ubiquitous light weight distribution system.

Since Esper can be compiled to a single Lua script, it can be included in the git repo of a project alongside a .esper file to give users an option to quickly install systems needed for the project, that might not be installable on their distribution. For example, Fennel is not packaged on Debian, so if you want to build a fennel project you need to go gather fennel before you can do anything.

My friend Jesse and I ran into this headache while trying to define the build steps for our project fa. While I can readily apk add fennel, and just move forward, Debian and RHEL users can't. It simply isn't packaged for their systems, and neither Jesse nor I have the resources to fix that. But we can stop gap it at least by providing Esper and the build script necessary for those missing dependencies.


{                                                                               
   fetch={url="https://git.sr.ht/~technomancy/fennel/archive/0.6.0.tar.gz",     
          git=false,                                                            
          outf="fennel-0.6.0.tar.gz",                                           
          extract=true,                                                         
          atype="gzip"},                                                        
   builddir="/fennel-0.6.0",                                                    
   depends={                                                                    
      alpine="lua5.3 lua5.3-dev lua5.3-libs gcc make",                          
      debian="lua5.3 liblua5.3 liblua5.3-dev gcc make"                          
   },                                                                           
   build={"make fennel-bin"},                                                   
   rename={                                                                     
      {old="fennel", new="fennel-lua"},                                         
      {old="fennel-bin", new="fennel"}                                          
   },                                                                           
   inst={                                                                       
      {perms=755, {"fennel"}, out="/usr/local/bin/"}                            
   }                                                                            
}
 

Above is the esperbuild for fennel 0.6.0. These are the only instructions needed to build and install fennel from the official source. And all the user needs to have installed is Lua to be able to run Esper itself. A simple table, to provide what should be a simple and legible format/methodology to install needed resources. In my mind, it's a little bit like having a portable AUR manager. Provided custom user made build scripts you can simply conjure up the package with Esper. This makes it easy to share interesting packages between friends, or build specific tool sets for internal development, or even deploy complex systems into docker containers.

Further Esper can be used to easily share internal tools in settings where engineers are spread across a heterogeneous amalgamation of devices, without having to redefine the build/installation for every device type.

Conclusion

Chenmark has been an absolute font of personal and professional challenge and self exploration that has resulted in an almost overwhelming amount of "I could do X"-ism. I've done plenty of package management, and happily toil building on anything and everything. So why not try and make my own build system which I can use truly anywhere?"

If Esper sounds interesting to you, I highly suggest you check out the Gitlab repo where there is some documentation on the esperbuild internals. Documentation is hardly complete, but I'm happy to answer questions should anyone be curious.

Bio

(defparameter *Will_Sinatra* '((Age . 31) (Occupation . DevOps Engineer) (FOSS-Dev . true) (Locale . Maine) (Languages . ("Lisp" "Fennel" "Lua" "Go" "Nim")) (Certs . ("LFCS"))))

"Very little indeed is needed to live a happy life." - Aurelius