XCVB, growth hormone for Lisp?
I'm glad to announce the first public release of XCVB, the eXtensible Component Verifier and Builder for Common-Lisp, a shift from ASDF. While the software is still very primitive and under-documented, it comes with a simple example of how I automatically converted Exscribe to be compiled with it. XCVB currently depends on cl-launch and asdf-dependency-grovel.
The documentation I wrote already has everything there is to say about the little that XCVB is and what it isn't yet, so I won't repeat it here. Instead, I will discuss the deeper rationale for XCVB.
It is common wisdom that a (programming) language can't be designed perfectly in one go, but instead has to be grown (and trimmed) over a period of time to fit the needs and capabilities of the actors involved in communicating (programs), whether humans or machines. Well, Lisp in general, and Common Lisp in particular, has got the growing bit just right, where most other languages just don't get it: with macros and reader-macros, you can effectively turn Common-Lisp into any language you like. To quote Guy L. Steele Jr., "If you give someone Fortran, he has Fortran. If you give someone Lisp, he has any language he pleases."
However, since in Common Lisp these work by irreversibly side-effecting the read-time, compile-time, run-time and coffee-time environments, and since there is (currently) no standard way to cleanly restrict the scope of such changes or specify which set of changes you're using, any extension you use makes such side-effects exponentially more likely to confuse other people and programs, to the point that if you want your code to be reused by other people, you must refrain from using such extensions altogether, and stick to the base language (which incidentally isn't all that portable, with plenty of unspecified, mostly uncheckable corner cases).
And so, while Common Lisp enables you to grow your own language, it makes it difficult for that language to be the same as anyone else's. You're either using Common Lisp without much in terms of extension, in which case it's a rather bad programming language as compared to all there is out there; or you use it with plenty of home-grown extensions, in which case you're growing your own autistic language that only you can use -- plus you have to do all by yourself the heavy lifting of making that language not suck.
That's where XCVB comes into the fray. XCVB is a mechanism to specify a set of language extensions to use, and to restrict the scope of the use of such extensions. In other words, it's the bottom half of a module system. Unlike a full-fledged module system, XCVB won't handle syntax, namespaces, hygiene in macroexpansion, or any such fancy topic. However, XCVB will provide you with enough sanity that you will hopefully be able to handle these on top of XCVB. The promise of XCVB is that each module will be separately compiled, that the compile-time side-effects required to compile one module won't escape to break other modules.
Of course XCVB will also handle the proper staging of compiling the required compile-time dependencies (e.g. macros and declarations) before to attempt compiling the modules that use those dependencies. It will also play well with your larger-than-Lisp build (Makefile), and will even allow you to parallelize your Lisp build in the marvelous new world of multiprocessors. In the future, it may even handle distributed compilation farms, etc.
All these features will I hope make XCVB an attractive proposition, at least when XCVB itself will have grown beyond the current prototype. But the deep purpose of XCVB is not incremental or distributed compilation, it is to make Common Lisp a proper platform to grow languages spoken by more than one person.