April 10th, 2012

Mining for data in Quicklisp with qlmapper

The code in Quicklisp represents a large amount of interesting data. It's difficult to systematically query it, but I've been trying to make it easier, and one step in that process is available as a library called qlmapper.

qlmapper is pretty simple; it can load an arbitrary Lisp file after loading each system in Quicklisp. Each system is loaded in a fresh SBCL instance, so code that inspects and reports things can work from a mostly-clean Lisp environment.

Here are some pieces of information you could gather and share:
  • What packages does a given system define?
  • What are all the packages defined in Quicklisp? What systems introduce conflicting package names?
  • What foreign libraries does a system load? What is the name of the Debian package name that provides that foreign library?
  • Everything that Manifest does
  • Does any code use nreconc or revappend? (Or, more generally, a CL-aware code search engine.)
  • What ASDF system definitions include :author/:description/:license metadata? Which system definitions need to add it?
  • Which projects lack a README file (or some variation thereof)?
  • Which projects don't build and why?

Here's a small example script that I just made for qlmapper:

(defpackage #:foreign-report
  (:use #:cl))

(in-package #:foreign-report)

(defun canonical-name (library-pathname)
  (let* ((name (file-namestring library-pathname))
         (end (search ".so" name)))
    (subseq name 0 end)))

(defun find-library (line)
  (when (and (search "r-xp" line)
             (search ".so" line))
    (let ((path-start (position #\/ line)))
      (when path-start
        (subseq line path-start)))))

(defun maps-table ()
  (let ((table (make-hash-table :test 'equal)))
    (with-open-file (stream "/proc/self/maps")
      (loop for line = (read-line stream nil)
            while line do
            (let ((library (find-library line)))
              (when library
                (setf (gethash (canonical-name library) table) library)))))

(defun foreign-mappings ()
  (let ((table (maps-table)))
    (loop for object in sb-sys:*shared-objects*
          for name = (sb-alien::shared-object-namestring object)
          collect (list name (gethash (canonical-name name) table)))))

(with-open-file (stream "~/foreign-libraries.sexp"                                  
                        :direction :output                                      
                        :if-exists :append                                      
                        :if-does-not-exist :create)                             
  (let ((mappings (foreign-mappings)))                                          
    (when mappings                                                              
      (print (list cl-user:*qlmapper-object-name*                               

I called it with this: (qlmapper:map-loaded-systems "~/foreign-report.lisp")

An hour later, it produced an interesting report of library usage. It's not perfect, but it's a start, and can be refined to provide more accurate and useful answers. Even in this raw form, I can tell some interesting things. For example, I can tell which libraries I had to build from source (no Debian package available) by checking for "/usr/local" in the results.

What other stuff would be fun to discover about the universe of Quicklisp code? What changes and improvements to qlmapper would make it even easier to discover?