Zach's Journal - Making a small Common Lisp project

I sometimes see people say they don't know where to start when they want to write something in Common Lisp. Here's what I do when I get an idea and I want to explore it with Common Lisp.

Open a new file in the editor. If I'm not already running Emacs, I'll start it up, run M-x slime, and open a new Lisp file. I usually put new projects in their own directory, since even small projects tend to grow into multiple files.

So, let's say it's a project to extract some info out of Apache combined log files. I'll call it stumpgrinder and open up ~/src/stumpgrinder/stumpgrinder.lisp.

Add and evaluate defpackage and in-package forms. I have a tiny elisp function that does this for me, based on the file's name. If I'm in stumpgrinder.lisp, it spits out:

(defpackage #:stumpgrinder
  (:use #:cl))

(in-package #:stumpgrinder)

I evaluate the defpackage form with C-c C-c, then start writing definitions after the in-package form.

Write the code. I never add top-level, script-style code. Everything goes into a definition. The definitions might include functions, classes, special variables, and constants. I C-c C-c definitions as I write them. If I want to run functions or look at variables, I switch to the REPL with C-c C-z and evaluate things.

Make it (re-)loadable. That means being able to go from a freshly started Lisp to a Lisp with the software loaded. If the program remains a single file that doesn't require any other libraries or files, there's nothing left to do. I can open the file in Emacs and use C-c C-k or just use (load (compile-file "stumpgrinder")).

Projects grow, though, and libraries are handy. For example, let's say stumpgrinder needs cl-ppcre to do some string matching. That might mean updating the defpackage form:

(defpackage #:stumpgrinder
  (:use #:cl #:cl-ppcre))

However, compiling and loading the file directly will result in an error like this:

The name "CL-PPCRE" does not designate any package.

If I didn't update the defpackage form to use cl-ppcre, but decided to use cl-ppcre: package prefixes instead, the first prefixed symbol would trigger an error like this:

;   READ failure in COMPILE-FILE:
;     READER-ERROR at some position on some stream:
;       package "CL-PPCRE" not found

I need to make sure cl-ppcre is compiled loaded before my file is compiled and loaded. The easiest way is to make a simple ASDF system file.

Making an ASDF file for a one-file project is easy. Here's the entire contents of stumpgrinder.asd:

(asdf:defsystem #:stumpgrinder
  :depends-on (#:cl-ppcre)
  :components ((:file "stumpgrinder")))

The :depends-on line will make ASDF compile and load cl-ppcre before compiling and loading any of the files in the system. I can just use ,load-system in the slime REPL or (asdf:oos 'asdf:load-op 'stumpgrinder) or (what I usually use, SBCL-specific) (require 'stumpgrinder), when I'm in the project's directory.

I've seen some people put ASDF commands directly into source files to load dependencies. While this will work (if you arrange it correctly), it means you can't make your new project an ASDF dependency of some future software without factoring it all out into a system file anyway. Making a system file is so easy that there's no reason not to do it up front.

What if the file gets too big, and should be split up? Say, for example, I want to put some string-related functions used in stumpgrinder.lisp into a file called string.lisp. Rather than having two files, stumpgrinder.lisp and string.lisp, I split things up into three files. package.lisp defines the stumpgrinder package, and stumpgrinder.lisp and string.lisp both start with (in-package #:stumpgrinder).

stumpgrinder.asd now looks like this:

(asdf:defsystem #:stumpgrinder
  :depends-on (#:cl-ppcre)
  :components ((:file "package")
	       (:file "string"
		      :depends-on ("package"))
	       (:file "stumpgrinder"
		      :depends-on ("package"
				   "string"))))

Finish it (sort of). While I'm working on a project, I usually have *package* set to the project package in the REPL, and just access all functions willy-nilly. Once things are nearly finished, though, I look at what the main interfaces of the project might be, and export them in the defpackage form. For stumpgrinder, that might look something like this:

(defpackage #:stumpgrinder
  (:use #:cl #:cl-ppcre)
  (:export #:process-logfile
           #:with-logfile
           #:hit-count
           #:*logfile-directory*))

When the project is in progress, I leave the system file in the project directory and always load it from there. But when I'm finished, I symlink the system file into a more global place in the central registry, like ~/.sbcl/systems/ so I can just start Lisp and type (require 'stumpgrinder) to load it up, regardless of the current directory. I can also use it in other systems:

(asdf:defsystem #:log-analysis
  :depends-on (#:stumpgrinder #:drakma)
  :components (...))

There are plenty of other things involved in really finishing something, such as adding documentation, tests, revision control, etc. The scope and intended audience for the project will determine how much extra stuff you need to do to truly consider it done.