Log in

Mining for data in Quicklisp with qlmapper

The code in Quicklisp represents a large amount of interesting data. It's difficult to systematically query it, but I've been trying to make it easier, and one step in that process is available as a library called qlmapper.

qlmapper is pretty simple; it can load an arbitrary Lisp file after loading each system in Quicklisp. Each system is loaded in a fresh SBCL instance, so code that inspects and reports things can work from a mostly-clean Lisp environment.

Here are some pieces of information you could gather and share:
  • What packages does a given system define?
  • What are all the packages defined in Quicklisp? What systems introduce conflicting package names?
  • What foreign libraries does a system load? What is the name of the Debian package name that provides that foreign library?
  • Everything that Manifest does
  • Does any code use nreconc or revappend? (Or, more generally, a CL-aware code search engine.)
  • What ASDF system definitions include :author/:description/:license metadata? Which system definitions need to add it?
  • Which projects lack a README file (or some variation thereof)?
  • Which projects don't build and why?

Here's a small example script that I just made for qlmapper:

(defpackage #:foreign-report
  (:use #:cl))

(in-package #:foreign-report)

(defun canonical-name (library-pathname)
  (let* ((name (file-namestring library-pathname))
         (end (search ".so" name)))
    (subseq name 0 end)))

(defun find-library (line)
  (when (and (search "r-xp" line)
             (search ".so" line))
    (let ((path-start (position #\/ line)))
      (when path-start
        (subseq line path-start)))))

(defun maps-table ()
  (let ((table (make-hash-table :test 'equal)))
    (with-open-file (stream "/proc/self/maps")
      (loop for line = (read-line stream nil)
            while line do
            (let ((library (find-library line)))
              (when library
                (setf (gethash (canonical-name library) table) library)))))

(defun foreign-mappings ()
  (let ((table (maps-table)))
    (loop for object in sb-sys:*shared-objects*
          for name = (sb-alien::shared-object-namestring object)
          collect (list name (gethash (canonical-name name) table)))))

(with-open-file (stream "~/foreign-libraries.sexp"                                  
                        :direction :output                                      
                        :if-exists :append                                      
                        :if-does-not-exist :create)                             
  (let ((mappings (foreign-mappings)))                                          
    (when mappings                                                              
      (print (list cl-user:*qlmapper-object-name*                               

I called it with this: (qlmapper:map-loaded-systems "~/foreign-report.lisp")

An hour later, it produced an interesting report of library usage. It's not perfect, but it's a start, and can be refined to provide more accurate and useful answers. Even in this raw form, I can tell some interesting things. For example, I can tell which libraries I had to build from source (no Debian package available) by checking for "/usr/local" in the results.

What other stuff would be fun to discover about the universe of Quicklisp code? What changes and improvements to qlmapper would make it even easier to discover?


April 2015

Powered by LiveJournal.com