Log in

No account? Create an account

Making a small Lisp project with quickproject and Quicklisp

A few years ago I wrote about how I make small Common Lisp projects. With the availability of Quicklisp and ASDF2, my process has changed quite a bit. Here's what I do lately.

Get a comfortable environment (done only once, not once per project). First, as part of setting up my CL environment, I install Quicklisp and add it to my SBCL startup file. That means downloading quicklisp.lisp and then running a couple commands:

(load "quicklisp.lisp")

After that, Quicklisp loads automatically every time I start Lisp, and I have more than 300 libraries available to add as easy dependencies of my project, if needed.

Since I use Emacs and really like slime, I also usually install quicklisp-slime-helper with (ql:quickload "quicklisp-slime-helper") as one of my first steps on a new system. Emacs isn't required to follow the project-creation steps below, though.

Second, Quicklisp includes ASDF2. I like to set up ASDF2 to scan a particular directory tree, ~/src/lisp/, for local systems. To do that, I create a config file named ~/.config/common-lisp/source-registry.conf.d/projects.conf that has this in it:

(:tree (:home "src/lisp/"))

With that file in place, I can add new projects to that directory tree, and after an (asdf:initialize-source-registry) the project's systems will be loadable with asdf:load-system. I can unpack tarballs, check things out from source control systems, or create new projects and they'll all be easily available for loading.

ASDF2's default setup also scans a directory called ~/.local/share/common-lisp/source/, so if you don't mind putting projects there, you can use that without any additional configuration.

With Quicklisp installed and ASDF2 configured, here are the steps I follow when I get an idea and I want to explore it in Common Lisp.

note In the following examples, I use pathnames like #p"~/foo/" to mean (merge-pathnames "foo/" (user-homedir-pathname)). Most CL implementations allow this shorthand, but if yours doesn't, you will need to use the full pathname, e.g. #p"/home/xach/foo/".

Create a project skeleton with quickproject and load it. Quickproject is part of Quicklisp, so if it's not already loaded, I can just use this:

(ql:quickload "quickproject")

For this example, I'll make a project called swatchblade that generates rounded-rectangle PNGs of a particular color, and makes it available as a web service with Hunchentoot. To create a project skeleton for the project, I use this:

* (quickproject:make-project "~/src/lisp/swatchblade/"
                             :depends-on '(vecto hunchentoot))

The last part of the directory name is used as the new project name. I could choose a different name by passing the :name option explicitly.

quickproject:make-project creates several files in the swatchblade directory:

  • package.lisp
  • swatchblade.lisp
  • swatchblade.asd
  • README.txt

It also adds the directory to your ASDF configuration, so you can immediately load the skeleton project and its dependencies:

(ql:quickload "swatchblade")

ql:quickload will automatically install required libraries if they're available in Quicklisp.

Write some code. I open the newly-created ~/src/lisp/swatchblade/swatchblade.lisp and start hacking.

I define variables with defvar and defparameter, functions with defun and defgeneric, macros with defmacro, classes with defclass, etc. As I write each one, I compile it immediately with C-c C-c and occasionally switch over to the REPL to run some code.

As I use symbols from other projects, I update the defpackage form in package.lisp to import symbols. For example, I might want to use several Vecto symbols without package prefixes, so I could do this:

(defpackage #:swatchblade
  (:use #:cl)
  (:shadowing-import-from #:vecto

I don't put any library management code directly into Lisp source files. If I decide to use more external projects, I edit swatchblade.asd and add to the :depends-on list, e.g.:

(asdf:defsystem #:swatchblade
  :serial t
  :depends-on (#:vecto
  :components ((:file "package")
               (:file "swatchblade")))

Reloading the system with ql:quickload will install (if necessary) and load any newly-required systems.

Reorganize. For small projects, sometimes a single file suffices. Most of the time, though, I end up splitting code up into multiple files. In this example, I might make a file called utils.lisp, a file called graphics.lisp, and a file called web.lisp, and update the system definition from this, the system automatically created by quickproject:

(asdf:defsystem #:swatchblade
  :serial t
  :depends-on (#:vecto
  :components ((:file "package")
               (:file "swatchblade")))
...to be something like this:
(asdf:defsystem #:swatchblade
  :serial t
  :depends-on (#:vecto
  :components ((:file "package")
               (:file "utils")
               (:file "graphics")
               (:file "web")
               (:file "swatchblade")))

When the :serial t option is present in the defsystem, files are compiled and loaded in order. You can get more complicated in expressing inter-file relationships, but I haven't found it worth the trouble. I just organize my files so that functions and macros needed in later files are provided in earlier files.

Reuse. With something like swatchblade, I would probably re-use it by starting Lisp, loading the project with ql:quickload, and running a function to start Hunchentoot with the swatchbade handler in effect. The final package definition might look something like this:

(defpackage #:swatchblade
  (:use #:cl)
  (:export #:start-web-server)
  (:shadowing-import-from #:vecto

The session then might look something like this:

* (ql:quickload "swatchblade")
loading output
* (swatchblade:start-web-server :port 8080)
Server started on port 8080.

With a project that is meant to be used more as a library, the package would likely have many more exports, and I would re-use it by passing it with the :depends-on argument of quickproject:make-project, e.g.:

(quickproject:make-project "~/src/lisp/whimsytron/" 
                           :depends-on '(swatchblade))

From there I can go back to the "Write some code" step and continue the cycle.

If I want to reuse a project as a standalone program I can run from the command-line, I use Buildapp.



“ASDF2's default setup also scans a directory called ~/.local/share/common-lisp/source/, so if you don't mind putting projects there, …”

You don’t have to: just make a symbolic link to your projects directory here.
ln -s ~/path/to/my/projects ~/.local/share/common-lisp/source/

Managing development of libraries

This is a very nice overview of how to start a project.

However, I just wonder how people usually manage the development of their own libraries. Specifically, when you need a "stable" version and a "development" version. For example, I have right now three personal libraries where I need to keep them "frozen" so that I can keep running some experiments for a project I am working on. But at the same time, I need to change/improve some of the code in these libraries for future projects.

Is is just a case of using the vcs (e.g., git), or having different system names (e.g., lib1-stable and lib1-dev), a combination of several things, or something else?


Re: Managing development of libraries

I would suggest using git's branches.



Nice work! I didn't know about the quickproject aspect of quicklisp.

Makes me wish I were making small projects instead of always laboring on one giant one.

-- Chris P
very useful post, thank you.


Excellent post Zach! I've been using quicklisp to simplify setup of my dev environment & quicklisp seems like another great addition to my toolkit. Way to go man.

Re: Brilliant!

Sorry, I meant quickproject seems like another great addition to my toolkit.



Thanks, this is really great. Do you have any advice for how to include tests? Everyone seems to do it differently and I don't understand the additional "perform" methods that are often included with the defsystems. It would be really great if you could describe the steps you would take to add a test system onto a project that you started with quickproject.

Re: tests?

I don't usually write tests so I'm afraid I can't offer any guidance.


finally i can startup a lisp project on my own, more or less understanding what's going on. thanks.

btw, is it possible to subscribe to your blog via RSS?

matus (simplynitaai)
http://xach.livejournal.com/data/atom should work for subscribing.

One package for each file?

Yes it works.

i have one question in regards to the "Reorganize." section of your post:

I have split the file into two and extended :components keyword paramater in asd file accordingy. Do i have to define a separate package for each file in package.lisp and add the (in-package ...) expression in each file?

Because when i tried to just use the same (in-package ...) as in the first file without defining any additional package i am getting an error when loading the system via quickload, telling me that the package of the second file name does not exist...

In my case it would be enough two have one package for both files.

Thank You

".config/common-lisp/source-registry.conf.d/projects.conf" on Windows 7

Hello Zach

Do you maybe know where the folder ".config/common-lisp/source-registry.conf.d/projects.conf" has to be placed in Windows 7?

Is this Lisp implementation or Quicklisp/ASDF specific?

Upto now i successfully installed emacs + slime + SBCL + quicklisp in Windows 7 and this is the remaining setup i need.


Re: ".config/common-lisp/source-registry.conf.d/projects.conf" on Windows 7

I'm not sure. I got so sick of that setup that I added a special directory to Quicklisp for local projects.

Check the "local-projects" folder in your Quicklisp folder, wherever it is. You can put project directories in there and they are automatically quickloadable.

Re: ".config/common-lisp/source-registry.conf.d/projects.conf" on Windows 7

Ok, thanks for the reply. Will try to look for the answer somewhere else.

The reason why i am asking is, that i store my source files via dropbox (may not be the best idea) and work on them from different environments.

Anyway, i just found a work-around solution: directory symbolic link (junction point in windows 7) which i am placing in the quicklisp local-projects directory. Seems to work ok.

Thanks a lot!!!

I forgot...

I forgot to ask whether, assuming that the "~/.config/common-lisp/source-registry.conf.d/projects.conf" file is still relevant for ASDF 3, if there is a variable that I can set to a different path. I am asking because I can't find this path in the source code of ASDF 3 (maybe because it is not used anymore).

Cheers again.
Hello Zach,

apparently my first question went lost, because I can only see my second incomplete one. I am sending it again.

First of all, thank you again for sharing these tips.

I had asked whether the "~/.config/common-lisp/source-registry.conf.d/projects.conf" file is still relevant for ASDF 3. My understanding is that it isn't, because I can't find any mention of a "projects" file in the "asdf.lisp" file. Instead, it seems that now you can customize the "asdf:*source-registry-parameter*" variable, which defaults to NIL, but I have customized it by pointing to "projects.conf" without effect (ASDF 3.1.3).

Any idea on how a similar functionality could be emulated with ASDF 3? It seems useful to have one's projects directory automatically scanned.

Thanks for your attention.
Yes, the config file is still applicable. ASDF scans for "*.conf" files, so you will not see that specific name ("projects") in the source code.
Thanks, Zach. However, I have tried it and it doesn't work. ASDF fails to find my packages with the error:

Module HELLO-WORLD was not provided by any function on *MODULE-PROVIDER-FUNCTIONS*.

Yet I can load my package with SLIME. The content of "projects.conf" is:

(:tree (:home "Dropbox/projects/"))

I load ASDF 3 first and then Quicklisp. Is that correct?

Anyway, do you have any idea of the purpose of the ASDF:*SOURCE-REGISTRY-PARAMETER* variable?

Thank you.

I don't recommend using REQUIRE, which is what prompts the error related to modules. Use (asdf:load-system ...) or (ql:quickload ...) instead.

That projects.conf looks ok. To be honest, I don't use this configuration technique any more, I just put things in ~/quicklisp/local-projects/ or add directories to ql:*local-project-directories*.

Loading ASDF and then Quicklisp should work fine.

I don't know anything about ASDF:*SOURCE-REGISTRY-PARAMETER*, sorry.

LiveJournal comments are not a great forum for discussions like this. You might want to post questions to Stack Overflow, which gets a lot more traffic and helpers.