Making a small Common Lisp project

I wrote an updated version of this advice in 2010. You can read it here.

I sometimes see people say they don't know where to start when they want to write something in Common Lisp. Here's what I do when I get an idea and I want to explore it with Common Lisp.

Open a new file in the editor. If I'm not already running Emacs, I'll start it up, run M-x slime, and open a new Lisp file. I usually put new projects in their own directory, since even small projects tend to grow into multiple files.

So, let's say it's a project to extract some info out of Apache combined log files. I'll call it stumpgrinder and open up ~/src/stumpgrinder/stumpgrinder.lisp.

Add and evaluate defpackage and in-package forms. I have a tiny elisp function that does this for me, based on the file's name. If I'm in stumpgrinder.lisp, it spits out:

(defpackage #:stumpgrinder
  (:use #:cl))

(in-package #:stumpgrinder)

I evaluate the defpackage form with C-c C-c, then start writing definitions after the in-package form.

Write the code. I never add top-level, script-style code. Everything goes into a definition. The definitions might include functions, classes, special variables, and constants. I C-c C-c definitions as I write them. If I want to run functions or look at variables, I switch to the REPL with C-c C-z and evaluate things.

Make it (re-)loadable. That means being able to go from a freshly started Lisp to a Lisp with the software loaded. If the program remains a single file that doesn't require any other libraries or files, there's nothing left to do. I can open the file in Emacs and use C-c C-k or just use (load (compile-file "stumpgrinder")).

Projects grow, though, and libraries are handy. For example, let's say stumpgrinder needs cl-ppcre to do some string matching. That might mean updating the defpackage form:

(defpackage #:stumpgrinder
  (:use #:cl #:cl-ppcre))

However, compiling and loading the file directly will result in an error like this:

The name "CL-PPCRE" does not designate any package.

If I didn't update the defpackage form to use cl-ppcre, but decided to use cl-ppcre: package prefixes instead, the first prefixed symbol would trigger an error like this:

;   READ failure in COMPILE-FILE:
;     READER-ERROR at some position on some stream:
;       package "CL-PPCRE" not found

I need to make sure cl-ppcre is compiled loaded before my file is compiled and loaded. The easiest way is to make a simple ASDF system file.

Making an ASDF file for a one-file project is easy. Here's the entire contents of stumpgrinder.asd:

(asdf:defsystem #:stumpgrinder
  :depends-on (#:cl-ppcre)
  :components ((:file "stumpgrinder")))

The :depends-on line will make ASDF compile and load cl-ppcre before compiling and loading any of the files in the system. I can just use ,load-system in the slime REPL or (asdf:oos 'asdf:load-op 'stumpgrinder) or (what I usually use, SBCL-specific) (require 'stumpgrinder), when I'm in the project's directory.

I've seen some people put ASDF commands directly into source files to load dependencies. While this will work (if you arrange it correctly), it means you can't make your new project an ASDF dependency of some future software without factoring it all out into a system file anyway. Making a system file is so easy that there's no reason not to do it up front.

What if the file gets too big, and should be split up? Say, for example, I want to put some string-related functions used in stumpgrinder.lisp into a file called string.lisp. Rather than having two files, stumpgrinder.lisp and string.lisp, I split things up into three files. package.lisp defines the stumpgrinder package, and stumpgrinder.lisp and string.lisp both start with (in-package #:stumpgrinder).

stumpgrinder.asd now looks like this:

(asdf:defsystem #:stumpgrinder
  :depends-on (#:cl-ppcre)
  :components ((:file "package")
	       (:file "string"
		      :depends-on ("package"))
	       (:file "stumpgrinder"
		      :depends-on ("package"
				   "string"))))

Finish it (sort of). While I'm working on a project, I usually have *package* set to the project package in the REPL, and just access all functions willy-nilly. Once things are nearly finished, though, I look at what the main interfaces of the project might be, and export them in the defpackage form. For stumpgrinder, that might look something like this:

(defpackage #:stumpgrinder
  (:use #:cl #:cl-ppcre)
  (:export #:process-logfile
           #:with-logfile
           #:hit-count
           #:*logfile-directory*))

When the project is in progress, I leave the system file in the project directory and always load it from there. But when I'm finished, I symlink the system file into a more global place in the central registry, like ~/.sbcl/systems/ so I can just start Lisp and type (require 'stumpgrinder) to load it up, regardless of the current directory. I can also use it in other systems:

(asdf:defsystem #:log-analysis
  :depends-on (#:stumpgrinder #:drakma)
  :components (...))

There are plenty of other things involved in really finishing something, such as adding documentation, tests, revision control, etc. The scope and intended audience for the project will determine how much extra stuff you need to do to truly consider it done.

Tags:

Comments

Re: Sweetness

The very basics of ASDF is quite simple ("name system", "declare other systems it depends on", "list component files, with dependencies"). The main gotcha that (occasionally) bite me is "ASDF definition files MUST be loaded". ASDF hooks into the data provided by the file-loading to find out (among other things) where the definition file is and uses this to figure out the full path to the component files.

Re: Sweetness

How does that bite you? Do you try to evaluate defsystem forms in the editor?

Re: Sweetness

Put point after, press C-x C-e, see ASDF (rightly) barf and SLIME throwing up an error dialog. Swear, do a ,load-system andf fail to redo for another few months.