You are viewing fare

eyes black and white

XCVB, growth hormone for Lisp?

I'm glad to announce the first public release of XCVB, the eXtensible Component Verifier and Builder for Common-Lisp, a shift from ASDF. While the software is still very primitive and under-documented, it comes with a simple example of how I automatically converted Exscribe to be compiled with it. XCVB currently depends on cl-launch and asdf-dependency-grovel.

The documentation I wrote already has everything there is to say about the little that XCVB is and what it isn't yet, so I won't repeat it here. Instead, I will discuss the deeper rationale for XCVB.

It is common wisdom that a (programming) language can't be designed perfectly in one go, but instead has to be grown (and trimmed) over a period of time to fit the needs and capabilities of the actors involved in communicating (programs), whether humans or machines. Well, Lisp in general, and Common Lisp in particular, has got the growing bit just right, where most other languages just don't get it: with macros and reader-macros, you can effectively turn Common-Lisp into any language you like. To quote Guy L. Steele Jr., "If you give someone Fortran, he has Fortran. If you give someone Lisp, he has any language he pleases."

However, since in Common Lisp these work by irreversibly side-effecting the read-time, compile-time, run-time and coffee-time environments, and since there is (currently) no standard way to cleanly restrict the scope of such changes or specify which set of changes you're using, any extension you use makes such side-effects exponentially more likely to confuse other people and programs, to the point that if you want your code to be reused by other people, you must refrain from using such extensions altogether, and stick to the base language (which incidentally isn't all that portable, with plenty of unspecified, mostly uncheckable corner cases).

And so, while Common Lisp enables you to grow your own language, it makes it difficult for that language to be the same as anyone else's. You're either using Common Lisp without much in terms of extension, in which case it's a rather bad programming language as compared to all there is out there; or you use it with plenty of home-grown extensions, in which case you're growing your own autistic language that only you can use -- plus you have to do all by yourself the heavy lifting of making that language not suck.

That's where XCVB comes into the fray. XCVB is a mechanism to specify a set of language extensions to use, and to restrict the scope of the use of such extensions. In other words, it's the bottom half of a module system. Unlike a full-fledged module system, XCVB won't handle syntax, namespaces, hygiene in macroexpansion, or any such fancy topic. However, XCVB will provide you with enough sanity that you will hopefully be able to handle these on top of XCVB. The promise of XCVB is that each module will be separately compiled, that the compile-time side-effects required to compile one module won't escape to break other modules.

Of course XCVB will also handle the proper staging of compiling the required compile-time dependencies (e.g. macros and declarations) before to attempt compiling the modules that use those dependencies. It will also play well with your larger-than-Lisp build (Makefile), and will even allow you to parallelize your Lisp build in the marvelous new world of multiprocessors. In the future, it may even handle distributed compilation farms, etc.

All these features will I hope make XCVB an attractive proposition, at least when XCVB itself will have grown beyond the current prototype. But the deep purpose of XCVB is not incremental or distributed compilation, it is to make Common Lisp a proper platform to grow languages spoken by more than one person.

Comments

(Anonymous)

The name is a tad difficult

It reminds me of http://xkcd.com/

(Anonymous)

Huzzah!

I've been looking forward to this. I hope you keep improving it, especially the documentation.

ASDF was one of those things that, as a Lisp programmer I use a lot, but only because I have to. It's under-documented, often confusing, occasionally unreliable, and has lousy error messages. In desperation, I even tried out mk-defsystem, only to find out that it's even worse.

If XCVB can (despite its even sillier name) manage to make packages in Common Lisp not suck, you will be my hero.

Re: Huzzah!

The name XCVB came in a 15 minute session with Spencer Brody when I had to give a code-name for the project in a hurry. I later backronymed it, and the name stuck.

XCVB isn't ready for primetime yet, but I hope that as I build up a "plausible promise" I will attract a community to enhance it.

(Anonymous)

But is that what we should be doing with Lisp?

Lisp is, without question, THE best language for personal productivity. It is intensely optimised for the single speaker of a language, the single programmer of an entire system - indeed, for creators of worlds. It is a language unique in its ability to allow a single programmer to vanish up their own rear end (in the best of ways) whilst still outperforming a hundred-strong team.

Strip it of that, and you lose the point of Lisp. There are lots of languages that encourage that old oxymoron, "team productivity" (teams, by their very nature, can never run at maximum individual efficiency - they need the bureaucracy precisely BECAUSE they're made up of individuals... team programming vs individual programming is somewhat akin to the scientific method vs direct divine revelation). They exist to ease the pain of bureaucracy at the expense of individual expressiveness - precisely the opposite of Lisp. They suck where Lisp excels, but they also excel where Lisp sucks.

If collectivism is truly the way of the future, if there is no longer any place for the lone wolf programmer in this world, then don't try to drag Lisp into a place it never wanted to be; just let it die, and resign yourself to a future made of Java, PHBs and fail.

It will always be with the individual, the awkward, the obstreperous, the disagreeable that Lisp will thrive. Let it do so. Leave the herds to their grassy commons; accept that the sacrifice of climbing mountains, aside from the risk of occasionally falling off, is that you climb alone. And derive pride, not shame, from writing exquisite poetry in a language nobody else understands - for isn't that why we become dissatisfied with existing languages in the first place?
If you don't want to collaborate on writing software with other people, XCVB will hardly force you to do it. Stay in your hole, nobody cares. Someday, you may understand the concept of "division of labor". In the meantime, why don't you build computers out of parts you made yourself out of raw materials you extracted or grew yourself?

As for the rest of us, we understand that individuals benefit from collaborating with each other, and that good collaboration tools (as opposed to bureaucracy) are enablers that allow each one to do what he could never have dreamt to do all by himself.

Note that in the Lisp family, the PLT Scheme community has no problem providing a separate compilation module system that offers all that XCVB promises and much more (except that they don't handle distribution and caching at this point). People casually develop new languages and dialects that integrate in the PLT virtual machine, and can mix and match between modules created in many such languages.

(Anonymous)

"that old oxymoron, "team productivity" (teams, by their very nature, can never run at maximum individual efficiency - they need the bureaucracy precisely BECAUSE they're made up of individuals"

It's no oxymoron. Teams have productivity. Just because each piece of a system isn't running at perfect efficiency doesn't mean it's meaningless to measure the whole.

"If collectivism is truly the way of the future, if there is no longer any place for the lone wolf programmer in this world"

Using Lisp doesn't preclude using libraries. In fact, they're far more powerful than libraries in other languages, since Lisp libraries can add syntax (like ITERATE).

If you don't want ad-hoc collaboration with other Lisp programmers, why stop with libraries? Drop all of Common Lisp, too -- no need for standard functions like REMOVE-IF-NOT, when each "lone wolf" can just implement it herself. We'll give everybody McCarthy's Lisp 1 just to spite Java PHBs!
any extension you use makes such side-effects exponentially more likely to confuse other people and programs, to the point that if you want your code to be reused by other people, you must refrain from using such extensions altogether, and stick to the base language

I find this part of XCVB's motivation rather weak.  it doesn't seem to me that the main reason for people's reluctance to use language extensions in their libraries is that those extensions side-effect the Lisp image.  it rather seems to me that language extensions are not widely used mostly because straying from the base language means fewer people would consider working on your code due to the increased cognitive load — and if the code is not for public consumption in the first place, then "autism" is not a problem at all.

now I'm not saying that your reasoning here is invalid, just that your rationale is not very convincing as stated.  perhaps a real-world example or two could clarify things.

(you also mention lack of "macro hygiene" and criticize the CL package system.  these things may not be pretty from the language-theoretical standpoint, but it looks like people tend not to find them problematic in practice, so mentioning those without giving concrete examples doesn't help your cause either).

My point!

I mustn't have been clear because that was exactly my point!

Those who want to write libraries reused by other people have to refrain from modifying the language too much, because their modifications diminish the amount of people who'll contribute to the library (if you depend on non-standard macros) or worse, those people who'll use it at all (if you depend on non-standard reader-macros).

Those who don't care for other people reusing their software don't have other people reusing their software, anyway.

(As for why hygiene matters -- once again, if you refrain yourself from building ambitious higher-order macro extensions that other people build macro extensions on top of, packages are good enough indeed. Once again, limitations in the language lead to limitations in the ambition of programmers.)

Re: My point!

Those who want to write libraries reused by other people have to refrain from modifying the language too much, because their modifications diminish the amount of people who'll contribute to the library (if you depend on non-standard macros) or worse, those people who'll use it at all (if you depend on non-standard reader-macros).

indeed.  but the reasons here are more social (having to do with the cognitive load introduced by the modified language) than technical, and I don't see how XCVB or any other piece of language infrastructure is going to solve that.

Re: My point!

My point is precisely that technical and social aspects are deeply linked in ways that are obvious when you think of it, yet are ignored by most people who usually see only one aspect and don't understand the other.

XCVB is solving the problem by shielding modules from compile-time side-effects required by other modules by compiling each module in its own Lisp image, therefore allowing one to not have to know anything about how other modules are built, only about the interfaces he cares for.

By contrast, ASDF and other variants of defsystem, with their model of non-deterministically side-effecting a shared Lisp image, force the programmer to master the exponentially complex interactions between all the compile-time side-effects of each and every module that is part of the system.

Hence XCVB reduces the cognitive load required for the combination of multiple modules, and will enable the delimited use of advanced metaprogramming that wouldn't be socially manageable with ASDF.
(bah, comment editing seems to be a paid feature.  oh well)

EDIT: I do understand why one needs compile-time/run-time side effect separation in order to support massive parallelized builds, say.  but practically speaking you just need "cfasl" support for that, right?

CFASLs

CFASLs are a speed optimization. With enough discipline and at a speed cost, you could do everything with FASL.

What is *necessary* for parallelization of compilation is lack of compile-time side-effects across modules.

Note that in practice, incremental compilation with ASDF *requires*, but does not *enforce* such meta-level purity. Which leads to incremental compilation not working on big projects. At ITA, most everyone got used to compiling our huge projects from clean, because incremental build *may* unpredictably break, or worse, yield a false positive that you think passes the regression test suite but really won't if you commit it and other people build from clean.

(Anonymous)

Side effects

So from what I read in your documentation, you still allow all the normal CL side-effects, you just make the user be more explicit about how to load them. Is that correct?

Re: Side effects

Yes.

Compile-time side-effects are saved (if CFASLs are available), then discarded (whether they are available or not) at the end of every file compilation.

Before any compilation happens, the declared side effects are loaded (compile-time if CFASLs are available, load-time otherwise - hence you MUST use (EVAL-WHEN (:COMPILE-TOPLEVEL :LOAD-TOPLEVEL) ...) around any form that is published for use at compile-time by other files), and *only* those declared side-effects. This makes the semantics of compilation deterministic and foreseeable.
eyes black and white

December 2014

S M T W T F S
 123456
78910111213
14151617181920
21222324252627
28293031   

Tags

Powered by LiveJournal.com