?

Log in

No account? Create an account

Making a small Lisp project with quickproject and Quicklisp

A few years ago I wrote about how I make small Common Lisp projects. With the availability of Quicklisp and ASDF2, my process has changed quite a bit. Here's what I do lately.

Get a comfortable environment (done only once, not once per project). First, as part of setting up my CL environment, I install Quicklisp and add it to my SBCL startup file. That means downloading quicklisp.lisp and then running a couple commands:

(load "quicklisp.lisp")
(quicklisp-quickstart:install)
(ql:add-to-init-file)

After that, Quicklisp loads automatically every time I start Lisp, and I have more than 300 libraries available to add as easy dependencies of my project, if needed.

Since I use Emacs and really like slime, I also usually install quicklisp-slime-helper with (ql:quickload "quicklisp-slime-helper") as one of my first steps on a new system. Emacs isn't required to follow the project-creation steps below, though.

Second, Quicklisp includes ASDF2. I like to set up ASDF2 to scan a particular directory tree, ~/src/lisp/, for local systems. To do that, I create a config file named ~/.config/common-lisp/source-registry.conf.d/projects.conf that has this in it:

(:tree (:home "src/lisp/"))

With that file in place, I can add new projects to that directory tree, and after an (asdf:initialize-source-registry) the project's systems will be loadable with asdf:load-system. I can unpack tarballs, check things out from source control systems, or create new projects and they'll all be easily available for loading.

ASDF2's default setup also scans a directory called ~/.local/share/common-lisp/source/, so if you don't mind putting projects there, you can use that without any additional configuration.

With Quicklisp installed and ASDF2 configured, here are the steps I follow when I get an idea and I want to explore it in Common Lisp.

note In the following examples, I use pathnames like #p"~/foo/" to mean (merge-pathnames "foo/" (user-homedir-pathname)). Most CL implementations allow this shorthand, but if yours doesn't, you will need to use the full pathname, e.g. #p"/home/xach/foo/".

Create a project skeleton with quickproject and load it. Quickproject is part of Quicklisp, so if it's not already loaded, I can just use this:

(ql:quickload "quickproject")

For this example, I'll make a project called swatchblade that generates rounded-rectangle PNGs of a particular color, and makes it available as a web service with Hunchentoot. To create a project skeleton for the project, I use this:

* (quickproject:make-project "~/src/lisp/swatchblade/"
                             :depends-on '(vecto hunchentoot))
"swatchblade"

The last part of the directory name is used as the new project name. I could choose a different name by passing the :name option explicitly.

quickproject:make-project creates several files in the swatchblade directory:

  • package.lisp
  • swatchblade.lisp
  • swatchblade.asd
  • README.txt

It also adds the directory to your ASDF configuration, so you can immediately load the skeleton project and its dependencies:

(ql:quickload "swatchblade")

ql:quickload will automatically install required libraries if they're available in Quicklisp.

Write some code. I open the newly-created ~/src/lisp/swatchblade/swatchblade.lisp and start hacking.

I define variables with defvar and defparameter, functions with defun and defgeneric, macros with defmacro, classes with defclass, etc. As I write each one, I compile it immediately with C-c C-c and occasionally switch over to the REPL to run some code.

As I use symbols from other projects, I update the defpackage form in package.lisp to import symbols. For example, I might want to use several Vecto symbols without package prefixes, so I could do this:

(defpackage #:swatchblade
  (:use #:cl)
  (:shadowing-import-from #:vecto
                          #:with-canvas
                          #:rounded-rectangle
                          #:set-rgb-fill
                          #:save-png-stream))

I don't put any library management code directly into Lisp source files. If I decide to use more external projects, I edit swatchblade.asd and add to the :depends-on list, e.g.:

(asdf:defsystem #:swatchblade
  :serial t
  :depends-on (#:vecto
               #:hunchentoot
               #:cl-colors)
  :components ((:file "package")
               (:file "swatchblade")))

Reloading the system with ql:quickload will install (if necessary) and load any newly-required systems.

Reorganize. For small projects, sometimes a single file suffices. Most of the time, though, I end up splitting code up into multiple files. In this example, I might make a file called utils.lisp, a file called graphics.lisp, and a file called web.lisp, and update the system definition from this, the system automatically created by quickproject:

(asdf:defsystem #:swatchblade
  :serial t
  :depends-on (#:vecto
               #:hunchentoot)
  :components ((:file "package")
               (:file "swatchblade")))
...to be something like this:
(asdf:defsystem #:swatchblade
  :serial t
  :depends-on (#:vecto
               #:hunchentoot)
  :components ((:file "package")
               (:file "utils")
               (:file "graphics")
               (:file "web")
               (:file "swatchblade")))

When the :serial t option is present in the defsystem, files are compiled and loaded in order. You can get more complicated in expressing inter-file relationships, but I haven't found it worth the trouble. I just organize my files so that functions and macros needed in later files are provided in earlier files.

Reuse. With something like swatchblade, I would probably re-use it by starting Lisp, loading the project with ql:quickload, and running a function to start Hunchentoot with the swatchbade handler in effect. The final package definition might look something like this:

(defpackage #:swatchblade
  (:use #:cl)
  (:export #:start-web-server)
  (:shadowing-import-from #:vecto
                          #:with-canvas
                          #:rounded-rectangle
                          #:set-rgb-fill
                          #:save-png-stream))

The session then might look something like this:

* (ql:quickload "swatchblade")
loading output
* (swatchblade:start-web-server :port 8080)
Server started on port 8080.

With a project that is meant to be used more as a library, the package would likely have many more exports, and I would re-use it by passing it with the :depends-on argument of quickproject:make-project, e.g.:

(quickproject:make-project "~/src/lisp/whimsytron/" 
                           :depends-on '(swatchblade))

From there I can go back to the "Write some code" step and continue the cycle.

If I want to reuse a project as a standalone program I can run from the command-line, I use Buildapp.

Comments

Hello Zach,

apparently my first question went lost, because I can only see my second incomplete one. I am sending it again.

First of all, thank you again for sharing these tips.

I had asked whether the "~/.config/common-lisp/source-registry.conf.d/projects.conf" file is still relevant for ASDF 3. My understanding is that it isn't, because I can't find any mention of a "projects" file in the "asdf.lisp" file. Instead, it seems that now you can customize the "asdf:*source-registry-parameter*" variable, which defaults to NIL, but I have customized it by pointing to "projects.conf" without effect (ASDF 3.1.3).

Any idea on how a similar functionality could be emulated with ASDF 3? It seems useful to have one's projects directory automatically scanned.

Thanks for your attention.
Yes, the config file is still applicable. ASDF scans for "*.conf" files, so you will not see that specific name ("projects") in the source code.
Thanks, Zach. However, I have tried it and it doesn't work. ASDF fails to find my packages with the error:

Module HELLO-WORLD was not provided by any function on *MODULE-PROVIDER-FUNCTIONS*.

Yet I can load my package with SLIME. The content of "projects.conf" is:

(:tree (:home "Dropbox/projects/"))

I load ASDF 3 first and then Quicklisp. Is that correct?

Anyway, do you have any idea of the purpose of the ASDF:*SOURCE-REGISTRY-PARAMETER* variable?

Thank you.

I don't recommend using REQUIRE, which is what prompts the error related to modules. Use (asdf:load-system ...) or (ql:quickload ...) instead.

That projects.conf looks ok. To be honest, I don't use this configuration technique any more, I just put things in ~/quicklisp/local-projects/ or add directories to ql:*local-project-directories*.

Loading ASDF and then Quicklisp should work fine.

I don't know anything about ASDF:*SOURCE-REGISTRY-PARAMETER*, sorry.

LiveJournal comments are not a great forum for discussions like this. You might want to post questions to Stack Overflow, which gets a lot more traffic and helpers.