I'm starting a movement to consolidate Common Lisp libraries. To participate, here's the ten point program summary:
- Pick your favorite problem domain
- Identify all libraries that address it
- Work with various library authors
- Pick the most promising library
- Declare it THE library
- Add missing features present in other libraries
- Declare other libraries obsolete
- Rename package to short name identifying domain
- Invite all users to migrate to new library
- Profit
Most of it is pretty straightforward, but below are some details.
Purpose
The whole point of the exercise is to build a stronger Common Lisp community, where people cooperate better, by being able to more easily use other people's code or have your code used by other people.
Background
At ILC'2012, there was a vast consensus that the Common Lisp library situation could be vastly improved. On the one hand, thanks to Quicklisp (with congratulations again to Xach), it is easy at long last to load a set of libraries with all their transitive dependencies. On the other hand, in the times before that was easy, we have developed a culture of non-sharing, whereby everyone reinvents his own libraries, each only covering the parts of the domain used by the author. Moreover, it is quite hard to navigate through the set of existing libraries and figure out which is the best one to use for a given problem, and so people choose at random, by cloning some system they've seen, or following the advice of a friend.
For a same problem domain, there may thus be more than a dozen different libraries, and each of them might cover only 80% of the domain at best, leaving users to fend for themselves for the 20% or more remaining unimplemented features. Of course, which 80% (or often, less than 50%) of the domain is covered by a particular library varies with the library, and the union of all libraries might still not cover 100% of what users desire. Olin Shivers once famously complained about this state of things in the world of Scheme, in the preamble to the documentation to his Scheme Regular Expressions library. The situation is no better in the world of Common Lisp today.
As an extreme case, I counted no fewer than 18 different unit test libraries in Quicklisp: (length '(lisp-unit xlunit unit-test fiveam stefil hu.dwim.stefil pcl-unit-test tap-unit-test cl-test-more monkeylib-test-framework ptester testbild xptest eos rt lift test-harness nst)) Cliki lists many more, and this still doesn't include countless library-less ad hoc solutions, nor does it count an unknown number of proprietary systems, of which at least four different ones have been developed at ITA. Do we need that many different unit test libraries? Is there even more than one different underlying design principle behind them?
Details
Each of the points in my 10-point program is important. If you skip one of them, you may just end up making things worse.
1- First, we can subdivide the work as building one consolidated library per problem domain. This means that this general consolidation project can be distributed among many people; it doesn't need to be done by a single person. Moreover, whoever works on consolidation in a domain should care enough about the domain to do the Right Thing™, rather than stop with yet another half-done job. Presumably, one of the existing library authors would be concerned enough to do it.
2- To fully solve the problem domain, we must identify all the libraries, and all their current and desired features, so that we may provide a 100% replacement to each and every one of these existing libraries, based on an architecture that makes all the future features possible.
3- It is extremely important to work with existing library authors. They are the ones who possess the expertise and the influence, that we want to cooperate. If for some reason there are irreconcilable differences that will lead to the survival of more than one library, these differences deserve to be documented, so that users can make their choice based on meaningful information. Sometimes, it might be possible for authors to layer their different approaches features on top of a common core library, or to build the common layer on top of a parameterized substrate that can offer the same functionality in different flavors: just because there is ultimate incompatibility at some level doesn't mean there cannot be code sharing at another level, after proper refactoring.
4- Out of the inspection of the existing code base, and the discussion with the various authors, one library should emerge as most promising, because its architecture is more expressive, because its maintainer is more responsive, because it sports more features, because its code is cleaner, because it has more a comprehensive test suite, because it has more enthusiastic users, etc.
5- By gathering a public consensus on which library to further develop as a 100% solution, our consolidators will already do a great service to the community: by directing the efforts of implementers and users alike into a single library, they will multiply the value of work poured into that problem domain, whereas that value was previously diluted over many libraries.
6- Often, it will appear that more work is required for the chosen library to satisfactorily replace the functionality of all the other existing libraries. That's fine. The consolidator, hopefully with the participation of the authors of previous libraries, will have to integrate all the demanded features in the common framework. This include bug fixes in corner cases, tests and documentation. Because these features were already written once, the difficulty is in integrating them, designing a nice interface to them, etc.
7- Even before the consolidated library is a 100% solution to the problem domain, it might be sufficient to replace some or all of the previous libraries. These libraries can then be declared officially obsolete, in their documentation, on cliki, on cl-user.net, in quicklisp, etc., with a link pointing to the new consolidated library. These libraries may well remain as historical documents, as personal toys, as backward compatibility tools, or as platforms for future experimentation. But hopefully, there should be no reason left for normal users to use anything but the consolidated library, until maybe one of the experiments succeeds and becomes the new best solution.
8- When the library is able to replace 100% of the functionality of all previous libraries, it may be renamed after the problem domain, and its package renamed accordingly. This will make it easier for users to find the library, and to understand read code that refers to the library. For instance, the future 100% solution to pattern matching would be called simply pattern-matching and its package would be PATTERN-MATCHING. Failing that, at least, the library should be properly indexed in all the above sites so anyone looking for pattern-matching in Lisp will easily find the library.
9- Now is the time to direct users to use the consolidated library and migrate their existing code. In odd cases, it may be necessary to fork a library that isn't maintained anymore, so it can be modified to use the new dependency. Or if there is not only no active maintainer but also no active user, the library might be declared obsolete, especially so if it is itself a candidate for being consolidated away.
10- When all this hard work is done, you can hopefully enjoy a better cooperating Common Lisp community, where it is easier to locate what library to use for a given goal, where the fewer active libraries are of higher quality, where your can pride in the code you write being more useful to more people, rather than being yet more effort thrown into yet another not-so-usable and not-so-used code base.
So there. Go talk to your fellow hackers and try to work together. A little communication goes a long way.
PS: After some research, I propose we use the clocc-devel mailing-list to discuss this Consolidation in general and synchronize our efforts, with the understanding that the technical discussion specific to each consolidated domain would be moved to the respective library's mailing-lists.
PPS: One year later, the only reported success is m2ym's optima library for pattern-matching, which I warmly recommend as a replacement for my own older fare-matcher, and for all previously published pattern-matching libraries for Common Lisp. On the other hand, at least one attempt at providing a unified testing library failed miserably at Step 3, only making things worse by adding yet another library (see obligatory xkcd picture below.) I'm a big fan of More Cooperation with Less Coordination when it's possible, but sometimes more coordination now is just required for more cooperation with less coordination later, dammit!
