April 9th, 2008

eyes black and white

Creationist programming vs Evolutionary programming, part VI

Here is the sixth installment in this series for my essay Creationist programming vs Evolutionary programming. Previous installments: Part I (Creationist programming, The Devil), Part II (Intelligent Design, Polytheism), Part III (Unintelligent Design, Lamarckism), Part IV (Supernatural Selection, Teleological Evolution), Part V (Natural Selection, Inside Evolution).

The Relevance of Paradigm Evolution

This evolution of programming paradigms is a nice story, but what is its relevance for software developers? After all, the tools described above already exist; they have been created, they have been engineered, they have been selected or they have emerged, without any of these paradigms being explicitly stated, much less used as a conscious guide. Do these paradigms correspond to anything real, or are they but a nice-sounding rationalization? Do we gain anything by spelling them out?

Well, as Daniel Dennett wrote, There is no such thing as philosophy-free science; there is only science whose philosophical baggage is taken on board without examination. This is true of computer science and computer engineering as of any other human endeavour. Just because you don't state your assumptions doesn't save you from the consequences of following them when they are erroneous, not anymore than putting your head in the sand would save you from predators you can't see. These paradigms do describe assumptions implicitly followed without a conscious decision, and each step in their evolution describes relevant phenomena to which earlier paradigms are blind. And those who make unconscious decisions are but surer victims of the problems they are blind to.

Realizing that some phenomena are not accidents happening during development, but constitute an essential part of it is necessary to properly address them. Failing to plan is planning to fail. If you assume say, the Intelligent Design paradigm, even though you may benefit from tools developed with latter paradigms, you will systematically waste resources trying to intelligently design what is beyond the reach of any intelligent design, or aiming at the only solutions reachable by it despite their being inferior to competition. However, if you go beyond intelligent design, you will come to better solutions naturally by letting them grow. By embracing a more primitive paradigm, you will introduce a lot of unnecesary nasty bugs by not taking seriously the systematic processes of weeding them out early with dedicated tools; you will systematically fail to consider cheap solutions that are at hand, but that do not lend themselves to a perfect algorithmic description, etc. Those who stay behind in terms of software development paradigm will be incapable of doing what will appear to them as clever lateral thinking, strokes of genius or unreachable fantasy, whereas those who master further paradigms will casually achieve feats previously deemed impossible by a simple systematized application of their more evolved paradigms.

To those who understand the relevance of such paradigms, the important open question is: what is the next paradigm, if any? Is the above Inside View of Evolution the be-all, end-all of programming paradigms? Is the refinement of existing tools our only hope? Or will some further paradigm catch on? Can one identify and adopt this paradigm early on, and thus get an edge over competition?

What then, if anything, is next on our road as far ahead of the current paradigm as that paradigm was of previous ones?

Paradigmatic Optimism

The simplest view about future paradigms is that there will be no new ones, at least none that works. Our understanding of software development is mature and as good as it can get as far as the big piture goes, though there may always be a myriad of minor details to get right. This is Present Optimism: the theory that we've already reached the limit of knowledge.

Of course, assuming there is finite understandable information about the big picture of software development, there will be diminishing returns in understanding the field and eventually not enough new relevant information to possibly constitute a new paradigm change for the better. And so we can be confident that this theory of Present Optimism will some day be true about software development paradigms as about many things.

On the other hand, considering how new the field of software development is and how fast it has changed in just the last few years, it seems premature to declare that we fully understand how software is developed and will not find new deep insights. If indeed our understanding of software development was to remain unchanged for, say, five to ten years, and all developers were to settle towards a finite set of well understood unchanging methods, then we could assert with much more confidence that indeed we have reached the acme of software development. But this hasn't nearly happened yet, and the case for Present Optimism is rather slim.

Another kind of optimism and a common idea about the future of software paradigms has always been that computers will somehow become more intelligent than men and will take over the menial task of programming, like djinns to whom you will give orders and who will grant your wishes. This is Extreme Future Optimism, or Millenarism: the theory that soon(er or later), we'll reach a Millenium where all our worries will be taken away.

However, this Optimism is based on a misunderstanding of what progress is about, a misunderstanding that is best dispelled by confronting it with the equal and opposite misunderstanding: the claim that such a future is bleak because it means machines will be taking all our jobs away. Hopefully the errors will cancel each other in a collision from which light will emerge.

Yes, computers in many ways have replaced humans for many tasks, and will replace humans for more tasks to come. The building of tools that replace human work in software development is what our whole story of paradigms was about. But competition by computerized tools does not destroy human jobs, it only displaces jobs towards new areas not covered by tools. Useful tools provide some of the same positive satisfactions as before and some more, while reducing the negative efforts; the goal of some previous jobs is fulfilled without the associated costs. The human resources previously used toward that goal are not destroyed but liberated; they are made available to be redirected to new useful endeavours that couldn't previously be afforded.

Furthermore, as long as humans and machines do not have the same relative performance in all activities, the law of comparative advantages ensures that the tasks relatively better done by meatware than by software will remain a domain of human activity. And even if machines do it better than humans, nobody prevents you from programming without machine help, or from choosing to sponsor a human rather than a machine for the programming tasks you need. Just like automation in other industries made these industries vastly more productive and mankind at large vastly more wealthy, so will automation in programming make software a more profitable industry and better serve mankind. Through all the software development tools already mentionned in the article above, automation already serves mankind, to a tremendous degree. Continuing to program in Java will no more provide job security than did programming in C++, COBOL or Assembly before; it will only guarantee a lot of wasted effort and ultimately failure in the Luddite refusal of automation.

What machines can neither possibly create nor destroy is on the one hand the desire for ever more, ever higher satisfactions, and on the other hand, the ability to adapt and work towards these satisfaction: in other words, human life, its drive and its spirit. Machines displace this life for the better, turning feats into chores, chores into menial tasks, menial tasks into assumable commodities. As our past worries are taken away, we worry about new often loftier tasks that become our focus. Ultimately, the only persons who create human jobs are human parents, and only illness and death destroy jobs away; the rest is a matter of organizing existing human resources. The fear of Artificial Intelligence is a lifeformist stance wrapped in the usual protectionist fallacies, and its narrowmindedness should inspire the same spite as racist or nationalist arguments before it.

Conversely, blind faith in Artificial Intelligence is yet another mystic superstition by millenarists dreaming of being saved from having to live their own lives. This blind faith is a cop out, in that it wishes away the very nature of programming and its intrinsic difficulties. Indeed, even if intelligent machines are to replace humans in the activity of programming, said machines won't be able cop out of a programming paradigm that way; the buck will have to stop somewhere, and the issues will have to be addressed. One of the main features of digital computer software as we know it is that it behaves, combines, and can be understood according to rigorous formal semantics in perfectly well-defined logics, whereas intelligence seems to be about dealing with fluid concepts, incomplete information, creative solutions, under misunderstood external pressures. Bridging that gap, if possible, can't be achieved by hand-waving. It requires a paradigm shift that the cop out precisely prevents from knowing.

The legitimate cop out is not to assume knowledge but to admit ignorance: my previous investigations didn't lead to any firm conclusion to this question, and I don't know have enough combined care for the matter and trust in the remaining available venues to investigate to afford further investigation. But are we reduced to this ignorance? Are there not things we know or can guess about the directions that the future may take?