You are viewing fare

eyes black and white

Consolidating Common Lisp Libraries

I'm starting a movement to consolidate Common Lisp libraries. To participate, here's the ten point program summary:

  1. Pick your favorite problem domain
  2. Identify all libraries that address it
  3. Work with various library authors
  4. Pick the most promising library
  5. Declare it THE library
  6. Add missing features present in other libraries
  7. Declare other libraries obsolete
  8. Rename package to short name identifying domain
  9. Invite all users to migrate to new library
  10. Profit

Most of it is pretty straightforward, but below are some details.

Purpose

The whole point of the exercise is to build a stronger Common Lisp community, where people cooperate better, by being able to more easily use other people's code or have your code used by other people.

Background

At ILC'2012, there was a vast consensus that the Common Lisp library situation could be vastly improved. On the one hand, thanks to Quicklisp (with congratulations again to Xach), it is easy at long last to load a set of libraries with all their transitive dependencies. On the other hand, in the times before that was easy, we have developed a culture of non-sharing, whereby everyone reinvents his own libraries, each only covering the parts of the domain used by the author. Moreover, it is quite hard to navigate through the set of existing libraries and figure out which is the best one to use for a given problem, and so people choose at random, by cloning some system they've seen, or following the advice of a friend.

For a same problem domain, there may thus be more than a dozen different libraries, and each of them might cover only 80% of the domain at best, leaving users to fend for themselves for the 20% or more remaining unimplemented features. Of course, which 80% (or often, less than 50%) of the domain is covered by a particular library varies with the library, and the union of all libraries might still not cover 100% of what users desire. Olin Shivers once famously complained about this state of things in the world of Scheme, in the preamble to the documentation to his Scheme Regular Expressions library. The situation is no better in the world of Common Lisp today.

As an extreme case, I counted no fewer than 18 different unit test libraries in Quicklisp: (length '(lisp-unit xlunit unit-test fiveam stefil hu.dwim.stefil pcl-unit-test tap-unit-test cl-test-more monkeylib-test-framework ptester testbild xptest eos rt lift test-harness nst)) Cliki lists many more, and this still doesn't include countless library-less ad hoc solutions, nor does it count an unknown number of proprietary systems, of which at least four different ones have been developed at ITA. Do we need that many different unit test libraries? Is there even more than one different underlying design principle behind them?

Details

Each of the points in my 10-point program is important. If you skip one of them, you may just end up making things worse.

1- First, we can subdivide the work as building one consolidated library per problem domain. This means that this general consolidation project can be distributed among many people; it doesn't need to be done by a single person. Moreover, whoever works on consolidation in a domain should care enough about the domain to do the Right Thing™, rather than stop with yet another half-done job. Presumably, one of the existing library authors would be concerned enough to do it.

2- To fully solve the problem domain, we must identify all the libraries, and all their current and desired features, so that we may provide a 100% replacement to each and every one of these existing libraries, based on an architecture that makes all the future features possible.

3- It is extremely important to work with existing library authors. They are the ones who possess the expertise and the influence, that we want to cooperate. If for some reason there are irreconcilable differences that will lead to the survival of more than one library, these differences deserve to be documented, so that users can make their choice based on meaningful information. Sometimes, it might be possible for authors to layer their different approaches features on top of a common core library, or to build the common layer on top of a parameterized substrate that can offer the same functionality in different flavors: just because there is ultimate incompatibility at some level doesn't mean there cannot be code sharing at another level, after proper refactoring.

4- Out of the inspection of the existing code base, and the discussion with the various authors, one library should emerge as most promising, because its architecture is more expressive, because its maintainer is more responsive, because it sports more features, because its code is cleaner, because it has more a comprehensive test suite, because it has more enthusiastic users, etc.

5- By gathering a public consensus on which library to further develop as a 100% solution, our consolidators will already do a great service to the community: by directing the efforts of implementers and users alike into a single library, they will multiply the value of work poured into that problem domain, whereas that value was previously diluted over many libraries.

6- Often, it will appear that more work is required for the chosen library to satisfactorily replace the functionality of all the other existing libraries. That's fine. The consolidator, hopefully with the participation of the authors of previous libraries, will have to integrate all the demanded features in the common framework. This include bug fixes in corner cases, tests and documentation. Because these features were already written once, the difficulty is in integrating them, designing a nice interface to them, etc.

7- Even before the consolidated library is a 100% solution to the problem domain, it might be sufficient to replace some or all of the previous libraries. These libraries can then be declared officially obsolete, in their documentation, on cliki, on cl-user.net, in quicklisp, etc., with a link pointing to the new consolidated library. These libraries may well remain as historical documents, as personal toys, as backward compatibility tools, or as platforms for future experimentation. But hopefully, there should be no reason left for normal users to use anything but the consolidated library, until maybe one of the experiments succeeds and becomes the new best solution.

8- When the library is able to replace 100% of the functionality of all previous libraries, it may be renamed after the problem domain, and its package renamed accordingly. This will make it easier for users to find the library, and to understand read code that refers to the library. For instance, the future 100% solution to pattern matching would be called simply pattern-matching and its package would be PATTERN-MATCHING. Failing that, at least, the library should be properly indexed in all the above sites so anyone looking for pattern-matching in Lisp will easily find the library.

9- Now is the time to direct users to use the consolidated library and migrate their existing code. In odd cases, it may be necessary to fork a library that isn't maintained anymore, so it can be modified to use the new dependency. Or if there is not only no active maintainer but also no active user, the library might be declared obsolete, especially so if it is itself a candidate for being consolidated away.

10- When all this hard work is done, you can hopefully enjoy a better cooperating Common Lisp community, where it is easier to locate what library to use for a given goal, where the fewer active libraries are of higher quality, where your can pride in the code you write being more useful to more people, rather than being yet more effort thrown into yet another not-so-usable and not-so-used code base.

So there. Go talk to your fellow hackers and try to work together. A little communication goes a long way.

PS: After some research, I propose we use the clocc-devel mailing-list to discuss this Consolidation in general and synchronize our efforts, with the understanding that the technical discussion specific to each consolidated domain would be moved to the respective library's mailing-lists.

PPS: One year later, the only reported success is m2ym's optima library for pattern-matching, which I warmly recommend as a replacement for my own older fare-matcher, and for all previously published pattern-matching libraries for Common Lisp. On the other hand, at least one attempt at providing a unified testing library failed miserably at Step 3, only making things worse by adding yet another library. I'm a big fan of More Cooperation with Less Coordination when it's possible, but sometimes more coordination is just required for more cooperation, dammit!

Comments

It sems to me that the various libraries are sometimes (perhaps often) justified by differences in features/API which are incompatible.

See for example: https://gitorious.org/com-informatimago/com-informatimago/blobs/master/common-lisp/html-generator/html-generators-in-lisp.txt
People go on using cl-who, despite/because an incompatible design and API.

Otherwise I am in favor, but we may very well end with N+1 libraries. It will still be good if we can point to the N+1th library to newbies.
Interestingly, this was the very first point I addressed in subsequent paragraphs of my original write up. But discussing the good and bad reasons and ways to merge or not to merge libraries soon led me to the dysfunctional aspects of the CL community, and that was not a productive thing to include in the present article. I wanted to focus on the positive actions that could often be taken.

So yes, once we narrow down what a library is about, we'll find that there is space for multiple solutions in a problem domain, each with a very different style. But still fewer than 18 of them, usually, and we still need focus to make one of them a 100% solution.
Good idea. I think part of the reason for the proliferation of libraries is that it's too much fun not to write your own! :P Seriously though, many promising libraries have been abandoned by their creators, so creating yet another library that serves the same purpose sometimes feels like the right decision at the time.

I think the Common Lisp community may also just be too opinionated to pick any one library & declare it to be THE library. eg. I think Clack is one of the most beautiful web libraries in any language, but WebLocks has a loyal following & it's difficult to even get people to reach agreement about what a web framework should be.

But I totally agree, we need to spend more time improving existing code instead of reinventing the wheel.
Yeah, good idea. I would like to see cl-emb updated and upgraded with improved documentation and new features. I think that it is a fairly good library for template-based text generation. Probably it is not THE library, but it is pretty much close to being one.

лисперы всех стран - объединяйтесь!

User ujeen referenced to your post from лисперы всех стран - объединяйтесь! saying: [...] Consolidating Common Lisp Libraries [...]
Hi Faré,

Thank you for your article.

What about having an umbrella project on top of the selected libraries?
It seems to me that the enterprise you describe implies some sort of supervision, at least to draw the consensuses. This could also allow some quality control on the testing / reliability / compatibility of the different libraries. It will also help maximize team work momentum.

To quote the tags of this article, consolidation, IMHO, won't happen without cooperation.


Camille
In the name of consolidating efforts, I want to reuse an existing mailing-list instead of creating a new one, if any applies. I remembered the existence of cl-gardeners, but said mailing-list is no more. Is there another one that could be used?
I once hear of cl-gardeners too, but I can't think of an active mailing list that I know except those of the various CL implementations (symptomatic ?). There is still the quicklisp users group, isn't it something that could be discussed with Xach?
And so the winner is... the clocc-devel mailing-list.
See you there, Faré.
what's wrong with c.l.l.?
(apart from the well known issues which are seemingly dire but which, amazingly, have failed to kill it so far).
eyes black and white

December 2014

S M T W T F S
 123456
78910111213
14151617181920
21222324252627
28293031   

Tags

Powered by LiveJournal.com