August 1st, 2007

Confidential to Bowie P. in Arizona

Bowie Poag hits TheDailyWTF.

The story is epic.

And it worked. After two and a half hours of hacking Perl, DB2, and NIS/YP, the system was back up. The group definitions were now broken into separate lines and the database was re-pushed across the network. Over the next week, Bowie learned enough Perl to totally replace Dave's jalopy of a script.

And still, as much as he would've loved to point the finger of blame on Dave, professional integrity prevented him from doing so. All of the blame fell squarely on Bowie's shoulders, and while he had some hurt feelings, it didn't hurt his career on the whole.

Wow! Unlike every other Bowie project, he did not blame someone else for its failure. Good show!

CPU-hog troubleshooting

I had a problem last night when my my website's server started sucking up all the CPU and spiking the load average.

The server is powered by tbnl, which uses a pool of threads to handle incoming requests. One of the threads was out of control, but the right one was hard to find. This snippet cancelled all pending requests (and caused a few errors) without shutting everything down:

(mapcar (lambda (thread)
	  (sb-thread:interrupt-thread thread
				      (lambda () 
					(throw 'tbnl:tbnl-handler-done
					       ""))))
	(sb-thread:list-all-threads))

To make it easier to find problematic threads in the future, I added this function to my handler setup:

(defun thread-name-wrapper (function path)
  (lambda ()
    (setf (sb-thread:thread-name sb-thread:*current-thread*)
	  (format nil "Handler for ~S [in progress]" path))
    (unwind-protect
	 (funcall function)
      (setf (sb-thread:thread-name sb-thread:*current-thread*)
	    (format nil "Handler for ~S [finished & idle]" path)))))

Now I can do this:

CL-USER> (sb-thread:list-all-threads)
(#<SB-THREAD:THREAD "Handler for \"/test\" [in progress]" {B9DBFC9}>
 #<SB-THREAD:THREAD "Handler for \"/signbot/\" [finished & idle]" {C24CDF9}>
 #<SB-THREAD:THREAD "Handler for \"/signbot/\" [finished & idle]" {DAFB531}>
 #<SB-THREAD:THREAD "Handler for \"/signbot/\" [finished & idle]" {C6D2F59}>
 #<SB-THREAD:THREAD "Handler for \"/signbot/\" [finished & idle]" {D9EE439}>
 #<SB-THREAD:THREAD "Handler for \"/roflbot/save\" [finished & idle]" {D93E541}>
 #<SB-THREAD:THREAD "Handler for \"/signbot/\" [finished & idle]" {C4793D1}>
 #<SB-THREAD:THREAD "Handler for \"/signbot/\" [finished & idle]" {C3DA011}>
 #<SB-THREAD:THREAD "repl-thread" {C5E4BF9}>
 #<SB-THREAD:THREAD "reader-thread" {C5E4AA1}>
 #<SB-THREAD:THREAD "control-thread" {C5E4961}>
 #<SB-THREAD:THREAD "auto-flush-thread" {C5D8B99}>
 #<SB-THREAD:THREAD "Swank" {BE160B1}> #<SB-THREAD:THREAD {B5E1BE1}>
 #<SB-THREAD:THREAD "initial thread" {B526899}>)

Yay!