Making a small Common Lisp project

I wrote an updated version of this advice in 2010. You can read it here.

I sometimes see people say they don't know where to start when they want to write something in Common Lisp. Here's what I do when I get an idea and I want to explore it with Common Lisp.

Open a new file in the editor. If I'm not already running Emacs, I'll start it up, run M-x slime, and open a new Lisp file. I usually put new projects in their own directory, since even small projects tend to grow into multiple files.

So, let's say it's a project to extract some info out of Apache combined log files. I'll call it stumpgrinder and open up ~/src/stumpgrinder/stumpgrinder.lisp.

Add and evaluate defpackage and in-package forms. I have a tiny elisp function that does this for me, based on the file's name. If I'm in stumpgrinder.lisp, it spits out:

(defpackage #:stumpgrinder
  (:use #:cl))

(in-package #:stumpgrinder)

I evaluate the defpackage form with C-c C-c, then start writing definitions after the in-package form.

Write the code. I never add top-level, script-style code. Everything goes into a definition. The definitions might include functions, classes, special variables, and constants. I C-c C-c definitions as I write them. If I want to run functions or look at variables, I switch to the REPL with C-c C-z and evaluate things.

Make it (re-)loadable. That means being able to go from a freshly started Lisp to a Lisp with the software loaded. If the program remains a single file that doesn't require any other libraries or files, there's nothing left to do. I can open the file in Emacs and use C-c C-k or just use (load (compile-file "stumpgrinder")).

Projects grow, though, and libraries are handy. For example, let's say stumpgrinder needs cl-ppcre to do some string matching. That might mean updating the defpackage form:

(defpackage #:stumpgrinder
  (:use #:cl #:cl-ppcre))

However, compiling and loading the file directly will result in an error like this:

The name "CL-PPCRE" does not designate any package.

If I didn't update the defpackage form to use cl-ppcre, but decided to use cl-ppcre: package prefixes instead, the first prefixed symbol would trigger an error like this:

;   READ failure in COMPILE-FILE:
;     READER-ERROR at some position on some stream:
;       package "CL-PPCRE" not found

I need to make sure cl-ppcre is compiled loaded before my file is compiled and loaded. The easiest way is to make a simple ASDF system file.

Making an ASDF file for a one-file project is easy. Here's the entire contents of stumpgrinder.asd:

(asdf:defsystem #:stumpgrinder
  :depends-on (#:cl-ppcre)
  :components ((:file "stumpgrinder")))

The :depends-on line will make ASDF compile and load cl-ppcre before compiling and loading any of the files in the system. I can just use ,load-system in the slime REPL or (asdf:oos 'asdf:load-op 'stumpgrinder) or (what I usually use, SBCL-specific) (require 'stumpgrinder), when I'm in the project's directory.

I've seen some people put ASDF commands directly into source files to load dependencies. While this will work (if you arrange it correctly), it means you can't make your new project an ASDF dependency of some future software without factoring it all out into a system file anyway. Making a system file is so easy that there's no reason not to do it up front.

What if the file gets too big, and should be split up? Say, for example, I want to put some string-related functions used in stumpgrinder.lisp into a file called string.lisp. Rather than having two files, stumpgrinder.lisp and string.lisp, I split things up into three files. package.lisp defines the stumpgrinder package, and stumpgrinder.lisp and string.lisp both start with (in-package #:stumpgrinder).

stumpgrinder.asd now looks like this:

(asdf:defsystem #:stumpgrinder
  :depends-on (#:cl-ppcre)
  :components ((:file "package")
	       (:file "string"
		      :depends-on ("package"))
	       (:file "stumpgrinder"
		      :depends-on ("package"
				   "string"))))

Finish it (sort of). While I'm working on a project, I usually have *package* set to the project package in the REPL, and just access all functions willy-nilly. Once things are nearly finished, though, I look at what the main interfaces of the project might be, and export them in the defpackage form. For stumpgrinder, that might look something like this:

(defpackage #:stumpgrinder
  (:use #:cl #:cl-ppcre)
  (:export #:process-logfile
           #:with-logfile
           #:hit-count
           #:*logfile-directory*))

When the project is in progress, I leave the system file in the project directory and always load it from there. But when I'm finished, I symlink the system file into a more global place in the central registry, like ~/.sbcl/systems/ so I can just start Lisp and type (require 'stumpgrinder) to load it up, regardless of the current directory. I can also use it in other systems:

(asdf:defsystem #:log-analysis
  :depends-on (#:stumpgrinder #:drakma)
  :components (...))

There are plenty of other things involved in really finishing something, such as adding documentation, tests, revision control, etc. The scope and intended audience for the project will determine how much extra stuff you need to do to truly consider it done.

Tags:

Comments

Sweetness

This is good stuff, I need to restructure my little Common Lisp project at home to use ASDF. I guess I need to dive in and learn ASDF and Slime, too.

Re: Sweetness

The very basics of ASDF is quite simple ("name system", "declare other systems it depends on", "list component files, with dependencies"). The main gotcha that (occasionally) bite me is "ASDF definition files MUST be loaded". ASDF hooks into the data provided by the file-loading to find out (among other things) where the definition file is and uses this to figure out the full path to the component files.

Re: Sweetness

How does that bite you? Do you try to evaluate defsystem forms in the editor?

Re: Sweetness

Put point after, press C-x C-e, see ASDF (rightly) barf and SLIME throwing up an error dialog. Swear, do a ,load-system andf fail to redo for another few months.
Good stuff. I wish something like this had been easy to find when I was starting out. Perhaps it would make a good addition to the Common Lisp Cookbook?

(Anonymous)

Jolly good stuff

For small projects you can get away with :serial t if you have a fairly straight forward set of file dependencies.

:components ((:file "package")
             (:file "string")
             (:file "stumpgrinder"))
:serial t


The most interesting part of the article though was the description of your approach to development. You never write top-level forms, always writing definitions instead. You use the REPL for inspecting stuff.

I tend to use the end of the current file I'm working on as a scratch pad full of top-levels while developing. As I get each form ironed out, I move it into a definition above the scratch pad area. It feels like molding a piece of clay into a vase and then setting it aside for firing. I almost never use the REPL in SLIME except to load system definitions, restart Lisp, change directories etc.

Re: Jolly good stuff

Good point on :serial t, but I don't tend to use it mostly because I'm afraid to work out the dependencies after the fact.

I use the REPL an awful lot. I use *, **, and *** a lot. I use the history a lot. I don't use slime-scratch buffers very much.

I'd be interested in seeing your mode of scratch form development in motion. (When I first saw slime screenshots, I thought it was no big deal, compared to ILISP. When I saw it in action, actually doing stuff, I was blown away...)

(Anonymous)

Re: Jolly good stuff

A great post Zach, worthy of addition into some published work!

I've written lisp both ways mentioned above in the last couple of months. It seems to depend on how complicated the 'top-level functions might be, when they are simple dolist two liners, then I don't write them in the source (until the end) and play a lot in slime REPL. When the 'top-level is complicated, I write it first and almost never use the REPL beyond running some check that a function makes the right type in the return. I don't think that there is any important difference between the two techniques.
This is a great guide, Thanks!

(Anonymous)

Good post. I rarely ever see useful introductory tips like this regarding Lisp.

Is it considered good form to always have an ASDF system file, even for small projects? I was looking at Luis Oliveira's ediware.lisp (http://common-lisp.net/~loliveira/ediware/ediware.lisp) and noticed that everything was defined in a single file.

Also, what's the purpose of (eval-when (:compile-toplevel :load-toplevel :execute) ...) in ediware.lisp (http://common-lisp.net/~loliveira/ediware/ediware.lisp)? It looks like that's used in-place of the ASDF system file.
This looks like a pretty specific project to have a standalone script file that could be loaded with something like --load. I don't write anything like that, personally.

(Anonymous)

hm, it seems like everyone who's writing Lisp is working on some sort of substantial project. I hardly ever run across simple functions/small scripts like I do with other languages.
Personally, I write a lot of small functions and scripts that fit into a larger Lisp web system. They just aren't a great fit for running from Unix command-lines.

(Anonymous)

curious

excellent post for us beginners.
now i am curious as to how do you organize all your small functions and scripts to work with slime. Just for example, your tiny little elisp helper function, how do you have it loaded into emacs?

mark

Re: curious

You can put it straight into your ~/.emacs if you like. I put it in a file called xach.el in my load-path and (require 'xach) in ~/.emacs to load it (along with many other little utilities).
Bookmarked.

Thanks for a very simple introduction to ASDF. Any other info I've found has been hard to get into, especially since I can't allocate much time to searching for a "for dummies" intro to system definition.

(Anonymous)

very nice

Very nice introduction, I appreciate it. Workflow is not often talked about.

Andy

(Anonymous)

one minor note

"or (what I usually use, SBCL-specific) (require 'stumpgrinder)"

Well, a minor modification to the CMUCL 'require gives the identical ability.
Thanks for posting this. It was very helpful to me. I'm working on my first project in Common Lisp, and I'm still just trying to find out how to work with the system. Previously, I just loading all my files in order by hand, which isn't so bad when you keep your REPL around, but the asdf solution is really what I was looking for.

It's great to read about how an experienced developer works. Many times in a new environment, I might find out how to accomplish something, but it'll irk me because it seems tedious or inelegant. However, if one could just look over the shoulder of an experienced developer, one could see which features to use where. There is some holistic knowledge that is rarely transmitted about these complicated tools. It's as though you're given a big bag of features each of which are documented, but the task of trying to thread them together to accomplish your own goals is entirely undocumented.

(Anonymous)

nice. thanks.

(Anonymous)

Good introductory article

Thanks a lot for this introduction to asdf'ing little projects. This was exactly the sort of problems I was dealing with in the past days as my lisp project got bigger and bigger and thus had to be split up into several files. At first I thought of adding some top-level code at the beginning of every file but soon realized that this wouldn't be a good idea. Hence I'm very glad to have stumbled upon your nice article.

(Anonymous)

Thank you

Very useful reminder, especially for new people.

-Lispnoob from #lisp
thanks, this is useful and concise :)

(Anonymous)

shameless plug

Here's my library used to provide IDE-like functionality for browsing ASDF systems:

http://tehran.lain.pl/stuff/asdf-browser.el

September 2014

S M T W T F S
 123456
78910111213
14151617181920
21222324252627
282930    
Powered by LiveJournal.com