Log in

No account? Create an account
eyes black and white

Software Irresponsibility

In the way of achieving a healthy software development environment, a lot of projects fall in one of these two DON'Ts: irresponsibility and territoriality. Irresponsibility is when there is no one in charge of making things right (with respect to a whole category of problems). Territoriality is when there is someone in charge, and he won't let anyone else touch the code without him. They may sound opposite to one another, but often irresponsibility is a result of territoriality, where the person in charge just isn't interested in the kind of problem you're experiencing, and so any consideration for such problems gets disregarded in favor of whatever fits the interests of the maintainer.

As a notorious example of irresponsible software design, I will cite PHP, whose maintainer has repeatedly claimed he was never interested in designing a good programming language. It shows. But I'm sure you'll find plenty of familiar examples in your own experience, from the irresponsive provider of some proprietary software you've been locked into using, to maybe someone from some other department of your company that you've had to deal with, and perhaps even to yourself with respect to your own users.

Of course, it is always possible to start a new project to replace the software you're dissatisfied with. But the costs associated to such endeavour are often huge. You will have to reimplement not just the parts that the replaced software did wrong, but also the parts it did right. While by assumption you can clearly see parts of the design that are wrong and do better with respect to those parts, you may be lacking the proficiency, time and interest to do as well on these other parts. Last but not least, the community around the original software, that made it valuable, will not switch to the new one for merely small gains, and the transition from the old system to the new system may impose demanding constraints of backwards compatibility.

Free Software can drastically reduce the cost of starting a new project, by allowing you to fork the existing code base, copy large chunks of it, or at the very least read the code of what you may have to be compatible with. It does not make the cost go down to zero, however. There again, cruft that made the original software bad enough that you felt like changing it will be a heavy burden upon you if you keep it, and a huge setback to reimplement if you decide to throw it away. And even in the world of free software, some licenses can incur a higher cost to forking than others: for instance, licenses designed to be incompatible with other licenses, licences that force you to distribute the code as source, that prevent you from making any adaptation and keep the name, or otherwise put hurdles in the way of forkers.

In the world of Common Lisp, an interesting case of not-so-responsible software is ASDF. Some of you might be tempted to sneer, is he going to blame the problem with ASDF on the original author who is long gone, and the current maintainer who isn't committing enough energy and availability to making the project go forward? No, not at all. Dan Barlow actually did a great job while he was there, and if no one is stepping up to offer more than Gary King provides, it's certainly not Gary who is to blame; he is getting blatant bugs fixed and accepting consensual patches, and at this point forking would be trivial to fork the git repo if anyone were dissatisfied with Gary's performance. Rather, I'll argue that any problem with ASDF is a symptom of the way responsibilities are distributed with respect to its update, and propose a small technical change that can hopefully improve things a lot.

When Dan Barlow wrote ASDF in 2001, he only wanted a better semi-portable CL build system that he could use on top of SBCL for his own projects, something that would be more declarative than the previous best alternative MK:DEFSYSTEM, and would only need to support modern ANSI CL implementations not old forgotten Lisp variants. In developing ASDF, he loosely followed the better design principles once proposed by Kent Pitman for an extensible CL build system. In all those objectives, ASDF was a tremendous success, and CL users should all be grateful to Dan for his achievement. However, as usual the problem lies with issues he didn't try to tackle.

Problems with ASDF include the painful way you have to configure it, deficient support for documentation or testing, broken support for conditional compilation, missing support for generated files, its weird TRAVERSE algorithm, etc. Unlike the issue of deterministic incremental compilation that prompted me to start XCVB, none of the above issues is unsolvable in itself within ASDF. But the constraints within which ASDF is developed make it hard to solve them. Happily, some of these constraints can be removed with a little bit of hackery.

Most importantly, ASDF has a distribution bootstrap problem: it specifies how to load other systems, but nothing specifies how ASDF itself is loaded into your system. That problem is left to whoever writes the Lisp implementation, distributes the Lisp package, or maintains a larger project within which ASDF is used. And in the world of Common Lisp, there are tens of different implementations, many of them packaged in various ways. To make the problem tractable, ASDF follows the constraint that it comes as a single file that may be loaded independently of its location, and doesn't depend on any third party library beside what is included by each supported implementation.

This means that the provider of libraries that use ASDF must rely on the least common denominator of ASDF features, unless he wants to force his users to make sure they upgrade their ASDF. Upgrading the current ASDF could conceivably be easy, but is actually hard, because of the inability to hot-upgrade a loaded ASDF. It is still possible to remove an old ASDF and replace it, and XCVB does it in an ugly way (see how xcvb/no-asdf deletes the ASDF package and anything that uses it); however, files compiled against the old ASDF may not work with the new one and vice versa (I've had "interesting" issues that way while using xcvb-master in slightly different setups).

This inability to upgrade further makes the pre-packaging of a Lisp implementation with a given version ASDF a bit of a damned if you do, damned if you don't proposition to Lisp implementers and distributors. Happily, implementers have the option of only providing ASDF when REQUIRE'd, and this is the preferred way for users to specify they want to use ASDF, when it is available. Distributors may not have this option inasmuch as there is no standard mechanism to hook into an implementation's REQUIRE. An example of a failed attempt at distributing Common Lisp with ASDF is common-lisp-controller. It tried to provide a way to make sure there is always a configured ASDF, but their solution only works if everyone uses only debian or another supported OS distribution, and only if the debian maintainers keep up with all the common-lisp software (implementations and libraries) that one may want to use; two hypotheses that have been disproved in practice with respect to the needs of many users.

To step back to the problem of responsibility, the current constraints of ASDF are such that a consistent upgrade of ASDF requires action on the part of tens of people, each in his own territory: ASDF contributors, Lisp implementers, package distributors. If only one hacker alone decides to upgrade ASDF, he has to incur the cost of the associated work of modifying ASDF and the risk that this will introduce incompatibilities, for little overall progress in terms of what users may rely upon unless he convinces tens of other people also do the same. Thus, the costs of adoption are high, the benefits of development are low, development is slow, and a cycle of stagnation continues.

If someone cares about the future of ASDF, I recommend they should address the upgradability bug, which will itself unlock a big hurdle on the way of further evolution of ASDF. Note that this upgradability includes both the technical ability to load a new ASDF on top of an old one, but probably also the proper configuration of ASDF, to load systems from a series of configured paths declared in user and system preference files.

Personally, I'd rather work on XCVB, and invite people interested in build systems to work on it instead of ASDF. But if XCVB is to prevail, it may as well prevail against the best ASDF that ASDF can be.

XCVB tries to avoid all the problems of ASDF by stepping back from the assumption that everything happens in a one Lisp world that has already been setup. Instead, XCVB will manage many Lisp worlds that it will help you setup. It doesn't live inside your Lisp image, and thus doesn't depend on Lisp implementers or distributors to provide it for you; it can therefore evolve fast without any issue due to slow distribution. It is also liberated from the constraint ASDF has of itself having no dependencies, because these dependencies would both make bootstrap more difficult and interfere with other versions of same software as part of the software being built. XCVB thus reuses plenty of libraries and is richer in features, and will use even more libraries in the future. All this in addition to its original essential distinguishing feature of providing deterministic incremental compilation.

XCVB isn't complete yet, but I believe that on supported implementations (currently only SBCL, CCL, CLISP, but I could add more on demand, or you could do it yourself), it is already better than ASDF. If you care for CL build systems, please fix ASDF; but please give XCVB a try, too.

Even if you don't care for CL build systems, organize your software developments so that every upgrade step can be done by the single maintainer of a single project, without requiring the coordination between many people in many different projects.

This lesson should be particularly dear to those who dream of making a Standard for Lisp, Scheme, Javascript, Java or any other language or piece of software: if you need to agree on something that will require future coordination between plenty of people, it is almost sure to fail to evolve fast enough to remain relevant, even though one or two iterations might make it live (see e.g. R6RS and its current attempted successor). Instead, you should offer software that already runs with a free software reference implementation that others can trivially adapt to their systems if needed (also in Scheme, compare the SRFI process). After your code has become de facto standard, you might not even care for it being any kind of de jure standard anymore. Whereas if there's going to be a de jure standard, it will be blood and conflict as long as there's anything to disagree about, and still resentment at the unsatisfying compromise that results.

The same idea also accounts for Conway's Law: if you divide your software project between many teams, each team will build its own code base, and the structure of the software will end up following the structure of the teams that build it. Parts of the software where the ownership is not clearly assigned but where authority is divided will be the scene for conflict, slow evolution, bad design (if design at all), etc. Vertical disintegration, where independent teams manage layers of the software and its life-cycle, will in particular lead to a multiplication of such dysfunctional interfaces between entrenched overlapping code bases the global architecture of which cannot be refactored.

Happily, solving the problem of irresponsible software is much easier than solving the problem of irresponsible government.

PS: I've since accepted co-maintainership of ASDF. If problems persist beyond March 2010, blame me.



Shortly after writing the above post, I became the de facto maintainer for ASDF.

We're in April 2010 already and problems persist, so blame me.

But there is an ASDF 2 ready in beta-test (currently 1.676), so not all is lost.

Meanwhile, Juanjo of ECL fame posted a flame about the existing ASDF practice:

And from there he started a discussion on comp.lang.lisp:
eyes black and white

January 2018



Powered by LiveJournal.com