April 24th, 2013

Quick and dirty progress tracking

Today I was running some analysis on about 9,000 files, basically mapping a function over each to warm up a cache. Something like this:

* (map nil 'analyze-file *9000-files*)
time passes

I had no idea how well it was progressing, and whether I'd need to take a snack break or let it run overnight. So I interrupted it and wrote a quick and dirty REPL utility function:

(defun :dot-every (count fun)
  (let ((i 0))
    (lambda (&rest args)
      (when (< count (incf i))
        (setf i 0)
        (write-char #\. *trace-output*)
        (force-output *trace-output*))
      (apply fun args))))

It prints out one dot per COUNT invocations of the function it returns, giving some indication of progress. Sensible values for COUNT depend on the volume of function calls.

For this problem, I called it with a COUNT of 100:

* (map nil (:dot-every 100 'analyze-file) *9000-files*)
............etc

The cached analyses printed out a ton of dots quickly, and the uncached analyses started printing dots at a slow but steady pace, and I could tell that it would be done in a few minutes instead of a few hours.

So now I'm going to use this to wrap up any function I have to call a ton of times and I want to get a sense of how it's progressing.