CPU-hog troubleshooting

« previous entry | next entry »
Aug. 1st, 2007 | 11:13 am

I had a problem last night when my my website's server started sucking up all the CPU and spiking the load average.

The server is powered by tbnl, which uses a pool of threads to handle incoming requests. One of the threads was out of control, but the right one was hard to find. This snippet cancelled all pending requests (and caused a few errors) without shutting everything down: 

(mapcar (lambda (thread)
	  (sb-thread:interrupt-thread thread
				      (lambda () 
					(throw 'tbnl:tbnl-handler-done
					       ""))))
	(sb-thread:list-all-threads))

To make it easier to find problematic threads in the future, I added this function to my handler setup: 

(defun thread-name-wrapper (function path)
  (lambda ()
    (setf (sb-thread:thread-name sb-thread:*current-thread*)
	  (format nil "Handler for ~S [in progress]" path))
    (unwind-protect
	 (funcall function)
      (setf (sb-thread:thread-name sb-thread:*current-thread*)
	    (format nil "Handler for ~S [finished & idle]" path)))))

Now I can do this: 

CL-USER> (sb-thread:list-all-threads)
(#<SB-THREAD:THREAD "Handler for \"/test\" [in progress]" {B9DBFC9}>
 #<SB-THREAD:THREAD "Handler for \"/signbot/\" [finished & idle]" {C24CDF9}>
 #<SB-THREAD:THREAD "Handler for \"/signbot/\" [finished & idle]" {DAFB531}>
 #<SB-THREAD:THREAD "Handler for \"/signbot/\" [finished & idle]" {C6D2F59}>
 #<SB-THREAD:THREAD "Handler for \"/signbot/\" [finished & idle]" {D9EE439}>
 #<SB-THREAD:THREAD "Handler for \"/roflbot/save\" [finished & idle]" {D93E541}>
 #<SB-THREAD:THREAD "Handler for \"/signbot/\" [finished & idle]" {C4793D1}>
 #<SB-THREAD:THREAD "Handler for \"/signbot/\" [finished & idle]" {C3DA011}>
 #<SB-THREAD:THREAD "repl-thread" {C5E4BF9}>
 #<SB-THREAD:THREAD "reader-thread" {C5E4AA1}>
 #<SB-THREAD:THREAD "control-thread" {C5E4961}>
 #<SB-THREAD:THREAD "auto-flush-thread" {C5D8B99}>
 #<SB-THREAD:THREAD "Swank" {BE160B1}> #<SB-THREAD:THREAD {B5E1BE1}>
 #<SB-THREAD:THREAD "initial thread" {B526899}>)

Yay!

Tags:

Link | Leave a comment | Add to Memories | Tell a Friend

Comments {3}

How?

from: anonymous
date: Aug. 4th, 2007 02:55 am (UTC)
Link

Very useful! But exactly how did you "added this function to my handler setup" ?

Reply | Thread

Zach Beane

Re: How?

from: [info]xach
date: Aug. 4th, 2007 11:30 am (UTC)
Link

I have a function that does something like this:

(setf (gethash path *handlers*) function)

Now it does this:

(setf (gethash path *handlers*) (thread-name-wrapper function path))

Reply | Parent | Thread

(no subject)

from: anonymous
date: Aug. 7th, 2007 08:16 pm (UTC)
Link

(defmethod hunchentoot:dispatch-request :around ((table t))
(let ((uri (hunchentoot:request-uri)))
(setf (sb-thread:thread-name sb-thread:*current-thread*)
(format nil "Handler for ~S [in progress]" uri))
(unwind-protect
(call-next-method table)
(setf (sb-thread:thread-name sb-thread:*current-thread*)
(format nil "Handler for ~S [finished & idle]" uri)))))

Works for me :); Thanks! - bhyde

Reply | Thread