François-René Rideau (fare) wrote,
François-René Rideau

Why Language Extensibility Matters

If you neglect some aspect of computation in designing your programming language, then the people who actually need that aspect will have to spend a lot of time dealing with it by themselves; and if you don't let them do it by extending your language, they'll have to do it with inarticulate barkings rather than civilized sentences.

So you think you can overlook such facts of life as having to support multiple concurrent isolated virtual worlds on a same machine, communicating with such other worlds on other machines? Forget to specify how concurrent activities can co-exist at all? Leave out persistence and retrieval of data outside of your language specification? Omit ways to interface directly with existing libraries and external programs? Or just do a shoddy job at it that is not robust or comprehensive enough to sustain serious use in software that matters to programmers?

Then programmers will retrofit into your language such crocks as threads, whereby user code may arbitrarily break the system invariants of your otherwise safe language (if at all). They will resort to brittle external tools such as shell scripts to bind together the pieces of your system. They will curse you as they scramble to reimplement the interfaces you left out, in as many different and differently fragile ways. They will waste countless months rebuilding infrastructure to properly talk to databases, and get crazy having to deal with persistence, transactionality and schema upgrades through manually coded conventions rather than automatically enforced mechanisms.

Now there are infinitely many possible aspects to programming, new ones being invented everyday and becoming popular every year; as a programming language designer, you can't be reasonably expected to have already provided a well-designed interface to all of them in your programming language. Handling all possible aspects of programming that anyone will ever imagine to care for not only requires more resources than you can ever provide, but also a wilder imagination than all you can ever have. And so, you can never design a system that does everything for everyone, all the time. Therefore there will be (infinitely) many things that your computing system won't be able to fully handle; even for the finitely many things that you handle, you will find that you lack the resources to update and maintain your system so it keeps handling those features as the related needs evolve.

But then, you should at least admit defeat, and do not try to fake having a solution: give your users direct access to the guts of the system with which they can build actual solutions, instead of providing any misdesigned partial "solutions" that can't possibly be made to fit their needs. Any half-assed pseudo-solution you offer will be treated as damage and be routed around. Programmers who need to get things done will bypass your puny file-system abstractions and directly use low-level system calls interfaces -- or they will just prefer better languages that don't introduce any superfluous impedance mismatch between them and the realities they have to deal with. Until of course such languages degenerate to the point that they do introduce an impedance mismatch once again! Which might be but a minor nuisance, or convince users to fork your language or migrate away, depending on how much that matters to them.

Your system can't be the be-all end-all for everyone all the time. That is why it is very important that your system should be extensible. Now, some systems are only extensible because the source code is available. That's already infinitely better than systems where the source code isn't. But one can do much better: you can design your system so that it can be extended from within, with some macro system as in Lisp or Scheme, or some grammar extension system as with camlp4 or pepsi.

The original Lispers had that principle that there should not be any system programmer privilege: any user of the language or system should be able to do anything that the system or language implementor can do. For instance, whereas in Pascal there was this magic multi-argument PrintLn procedure when all user-defined procedures had a fixed number of arguments, Lisp goes through great length so that all the magic available in any of its functions or special forms could just as well be reimplemented or done differently by language users with the available primitives. You can implement your own elaborate object system on top of the language, and that's even how the reference implementation of the Common Lisp Object System is itself written; that's also how in practice users implement say an automatically prefixing variant of with-accessors with the help of symbol-macros and/or setf-expanders.

Now when everything can be defined and redefined, there is the worry that you can't trust anything, because the most basic things might have been redefined. The worry is actually quite legitimate, as interesting malicious Javascript cross-site scripts have shown. But the solution is not to prevent language modification; the solution is to allow the user to abstract over and control which modified language variant he is using. Thus, anyone should be able to easily define a new language, create compilers, analyzers and processors for such a language, write programs in that language or program-generators targetting it. Any invocation of a program would make include the implicit or explicit specification of an appropriate processor, with sane defaults that can't cause unrelated programs to go wrong. This the Common Lisp community failed to do so far with its paradigm of side-effecting the read-time or compile-time environments. But PLT Scheme handles such language abstraction like a charm by letting each module declare what language it is written in.

To paraphrase Hayek, IndividuaLisp is thus an attitude of humility before the program languaging process and of tolerance to other opinions, and is the exact opposite of that intellectual hubris which is at the root of the demand for comprehensive direction of the program languaging process.

Tags: en, extensibility, humility, lisp, programming languages, tao of programming, tunes
  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded