Xmas wish: generate depends.mk from .asd file

Touting a little Xmas wish into the snowy intertubes: Many kudos to anyone who writes a little tool to generate a depends.mk Makefile from an .asd file. (Similar to gcc's -M / -MM flags.)

A Conditional Situations Function API Pattern

I haven't written for a long time, and before Zach is going to weep my occasional mumblings from Planet Lisp, I thought I better sit down and share some API pattern that I have grown fond of.

The pattern addresses conditional code paths in a function's execution, in particular exceptional situations. Typically, there are the following ways you want to deal with a conditional situation:

• return a caller-specific value,
• signal an error as a structured way to transfer control,
• signal a warning or log a message and continue execution,
• signal a warning or log a message and return from execution,
• ignore the conditional situation and just continue execution,
• or perform some arbitrary caller-specific action.

Using the pattern, we will be able to express all those ways conveniently and concisely in the call to the function.

As illustrative example throughout this posting, we will use JOIN-THREAD. (It was the recent update of SBCL's JOIN-THREAD that reminded me on this pattern.)

JOIN-THREAD waits until the passed thread finishes its execution and returns the values of evaluating the last form in the thread. There are two exceptional situations involved:

1. as one does not necessarily want to wait forever, a caller can specify a timeout that might expire;
2. the thread might not have finished gracefully.

The pattern I want to introduce would make it have the following function interface:

(join-thread thread &key timeout on-timeout on-failure)

Where ON-TIMEOUT and ON-FAILURE can be either functions taking a condition object, or a non-function value that would essentially be interpreted as if (CONSTANTLY <the-value>) was passed.

The code invoking the ON-TIMEOUT callback might be written as follows:

(if (functionp on-timeout)
    (return-from join-thread (funcall on-timeout <timeout condition>))
    (return-from join-thread on-timeout))

Notice that the callback is invoked with a Condition object. Notice further that it's invoked in a tail position of the function being defined. This is true for exceptional situations; in the broader case of conditional situations, the function will usually continue further after having invoked the callback. (Passing a non-function value should result in returning that value even in the case of a mere conditional situations; it makes writing tests much easier that purport to exercise exactly that code path.)

Despite being an exceptional situation (i.e. just continuing is usually impossible), we might still want to provide the caller the ability to proceed further by using Common Lisp's restart system. E.g. to allow a caller to just plainly continue and hang on waiting on the thread forever, we could have written:

(if (functionp on-timeout)
    (with-simple-restart (continue "Ignore timeout, hang on thread ~S." thread)
      (return-from join-thread (funcall on-timeout <timeout condition>)))
    (return-from join-thread on-timeout))

And we can already express a variety of different behaviours. To simply return :TIMEOUT when the timeout expires, we call JOIN-THREAD as follows:

(join-thread <thread> ... :on-timeout ':timeout)

To signal an error on timeout, we call JOIN-THREAD as follows:

(join-thread <thread> ... :on-timeout #'error) ; this should probably be the default

To turn the timeout into a mere warning, it is unfortunately not enough to simply pass #'WARN because WARN is specified to require a condition of subtype Warning.

(join-thread <thread> ... :on-timeout #'warn)  ; caveat: won't do

This restriction on WARN is superfluous but it's there, and we can always define our own variant:

(declaim (inline alert))
(defun alert (datum &rest args)
  (let ((condition (apply #'coerce-to-condition datum args)))
    (if (not (typep condition 'warning))
        (cl:warn "~A" condition)
        (cl:warn condition))))

(join-thread <thread> ... :on-timeout #'alert)

Notice that the above call will display a message on timeout and return NIL. To display a message (or log it using some arbitrary log mechanism) and go on waiting, we could use the following functions:

(declaim (inline invoke-and-continue))
(defun invoke-and-continue (function condition)
  (funcall function condition)
  (continue condition)
  (error "BUG: could not find a CONTINUE restart for condition ~S." condition))

(defun alert continue (condition)
  (invoke-and-continue #'alert condition))

(defun logg continue (condition)
  (invoke-and-continue #'logg condition))

(join-thread <thread> ... :on-timeout #'alert continue)
(join-thread <thread> ... :on-timeout #'logg continue)

Another example would be ALERT RESIGNAL which will resignal the condition. This can be useful during debugging when you want to display a condition's underlying message stemming from a certain function call that a handler higher up might catch -- just add :ON-TIMEOUT #'ALERT RESIGNAL to the call.

If desired, one could add

(defvar *timeout-behaviour* #'error)
(defvar *failure-behaviour* #'error)

and make :ON-TIMEOUT and :ON-FAILURE default to these variables.

So what's nice about this pattern?

After all the CL equivalent would be

(handler-bind ((timeout #'alert continue))
   (join-thread <thread> ...))

or

(handler-case (join-thread <thread> ...)
   (timeout () :timeout))

which may not be as concise but surely is not overly verbose either.

A couple of things.

For one, it's essentially Continuation Passing Style for exceptional situations which can be exactly what you want occasionally. Also the return-a-value case does not involve signaling a condition or an extra function call - which can make a difference. Likewise, handling an exceptional situation does not involve a non-local exit; just a function call and a local exit.

And another thing, it serves as a window of opportunity to document the exceptional situations in the function's docstring, and of course, it helps remembering these situations during automatic lambda list display.

And it's easier to change between different behaviours as it does not require changing a HANDLER-BIND to a HANDLER-CASE or vice versa, or having to add a BLOCK for explicit transfer of control out of a handler.

And, as mentioned before, the conciseness of the return-a-value case is very handy when writing tests.

It also makes code read more like prose. (He said, handwavingly^Wsternly.)

Use case for Restart-Bind

It's always a good day when you find a perfect use case for one Common
Lisp's less used operators. Lately I had the pleasure to find a good
opportunity for RESTART-BIND.

I've written a network simulator in a couple of hundreds lines of
Common Lisp (not counting our basic protocol and miscanellous stack)
to emulate mobile IP networks. We use it to review that our mobile
broadband accelerator is behaving the way we want it to behave. We
also use it in a complete virtual setup using UML instances for our
test suite as well as for the lab in our office.

For determinism purposes I wanted to be able to reseed the
*RANDOM-STATE* to its initial value by some means. And it turned out
that RESTART-BIND is just the right thing for that job:

(defun simulator (... &key (seed #xDEADBEEF) ...)
 (let ((*random-state* (sb-ext:seed-random-state seed)))
   (restart-bind
       ((reseed #'(lambda ()
                    (setf *random-state* (sb-ext:seed-random-state seed)))
         :report-function
           (formatter "Reseed *RANDOM-STATE* with initial :SEED value.")))
     ...)))

You can reseed manually by interrupting the network simulator and
using the RESEED restart, or you can reseed programmatically (e.g.
periodically after a certain time of inactivity) by

(invoke-restart 'reseed)

Can you feel that warm and fuzzy feeling? Just the right thing. :-)

PS.

Other fun note about restarts and RESTART-BIND. If you squint your
eyes, you will discover that restarts are essentially nothing else
than dynamically scoped local functions, and RESTART-BIND is basically
DYNAMIC-FLET.

What I'm up to lately

Long time no update.

For the last few months I've become rather silent not only on the planet.lisp blogosphere but pretty much on the whole Common Lisp open source world. The reason for that is that since the beginning of the year I've become proud part of a hot telecom startup. Right after having had a big rush to finish my bachelor thesis in one go.

I've spent the last three months hacking in Malaysia, mostly in Kuala Lumpur, though sugared with occasional visits to tropical islands to keep sanity above a reasonable threshold. I now just arrived on the West coast of Sweden (reasonably close by to Gothenborg) where we're going to spend the next month at as we're told August to be lovely around there.

You bet what we're hacking in. Yes, that's right, we're doing funky TCP/IP related optimization and analysis in that dead, slow language. And I can tell you it's marvellous! It's a whole different experience if you can just make use of the language without being constrained like you usually are when developing open source libraries. We don't have to try to keep the number of dependencies small. We don't have to care about package name collisions. For example, we have a package named FMT containing functions to be used in format strings via ~/FMT:FOO/. Common Lisp is definitively a very nice language for actual product development. And it shows that is has been used for that while it was designed.

Of course, there's more that makes my time so incredibly marvellous: having pleasure to work with bright and, I think, famous guys, like that pseudo-german hacker poster boy Luke Gorrie, the omniscient Stelian Ionescu, homeless dude Ties Stuij of stix.to fame, the Erlang celebrity Sean Hinde, business genius Jane Walerud, and That Clever Statistics Guy. Also seeing different parts of the world, meeting kind people of different cultures, and developing sellable products and thus having customers to care about.

Fun.

Lots of it!

Slime tidbits (2010-03-06): Slime & ECL

Together with Juan Jose Garcia-Ripoll, I worked on improving the integration of ECL into Slime.

He did a tremendous job on the ECL side adding all the stuff that's needed for a well-working swank backend, so if you're going to try out ECL with Slime, and are pleasantly surprised, please direct most of the gratitude towards him.

I threw out backwards compatibility, and you really have to get the 10.3.1 release otherwise the CVS of Slime will refuse to compile. There's no point in artificially maintaining backwards compatibility for something which barely worked.

Most noteworthy is the fact that you can now M-. all the way down, that is not just into the Lisp source base of ECL, but also into the C source base. And because it's based on TAGS file, M-. (and M-*) will continue to work once in a .c file. For illustration, I created an animated screenshot. (Blogspot seems to convert uploaded images into the PNG format which is the reason that I have to externally link to animated gifs rather than include them into by blog posting proper.)

There's of course still stuff to do:

• storing arglist information for user-written functions
• storing source-location information for each method of a gf
  
• introspection into C objects

Especially the last thing would be rather cool. And while I think the first two items are on Juanjo's agenda, the last item needs some brave hero looking for fame and glory. In case you want to volunteer, drop a mail to the ECL mailing list!

Slime tidbits (2010-03-05)

In December 2009, Stas Boukarev and myself added some really cool stuff to the slime-asdf contrib, and while I've always wanted to blog about it, I just haven't come around doing so.

• M-x slime-load-system (,load-system): Compile and load an ASDF system; that command currently hooks into the Slime compilation-notes machinery, so compilation notes, warnings, etc. will be collected and will end up in the *SLIME Compilation* buffer. This command has always been there, I just mention it for sake of completeness.

• M-x slime-open-system (,open-system): Open all the files specified in the system.

• M-x slime-isearch-system and M-x slime-rgrep-system: Run the command isearch and rgrep respectively on all the files specified in a system. That is particularly useful if slime-edit-definitions (M-.) and slime-edit-uses (M-?) won't do.
  
  In case of slime-rgrep-system, the commands next-error and previous-error will jump through the matches in the *grep* buffer. I bound those commands to F11 and F12, though by default they're also bound to M-g p and M-g n (also C-x `).
  

• M-x slime-query-replace-system: Poor man's refactoring tool; run query-replace on all the specified files in a system. See this animated screenshot for an exemplary run. (Animated gif was generated on behalf of Zach Beane's Skippy library!)

• M-x slime-query-replace-system-and-dependents: Like the former function, but also run query-replace on all the files of all systems depending on the user-queried system.

The latter two functions are really useful if you're past the initial state of something -- your system already grew to multiple files -- but still away from finishing and so you often want to rename identifier and slightly change APIs.

Munich Lisp Talk, 10th November 2009

Last Tuesday, I gave short talk about a library I have been writing on: Sequence Iterators.
The library is perhaps a bit misnamed because at the moment, it's really a convenience layer on top of iterators. It's supposed to provide tools to make writing functions that operate on sequences convenient, and yet not blatantly inefficient.

I'd be very much interested in feedback regarding rough corners if you find a case to apply the library. If you want to give it a shot, there's a file "more-sequence-functions.lisp" which contain prototypes of useful sequence function for which I haven't yet had the time to write them.

Please notice that the library is still work in progress.