This was written in 1993 at the conclusion of a semester of evolutionary theory. It is reproduced here particularly for what it says about the importance of variation vis a vis selection over evolutionary history.
Analogies frequently date themselves into what should be irrelevance, but which often is not. Understanding and misunderstanding of the evolution-defining mechanisms of variation and selection responsible for The Origin of Species have informed and misinformed many fields of study. In the process of proposing that products of extended evolution will show an enhanced capacity for innovation, this essay will look at the development of knowledge of biological evolution and various analogies which have been claimed to be based on that knowledge. The analogies to be refered to are intended to be rather less notorious than the mainly Eurocentric view of evolutionary directionality which was taken as justification for various since-condemned excesses.
The central thesis of this essay is that an extended history of variation and selection will 'metaselect' for capacity to innovate. As evidence for this thesis we include the biochemical evidence for complex dynamics of the eukaryotic genome and the palaeontological evidence for extinction events being the dominant selector of higher taxa. From that evidence it will be argued that it is useful to describe a significant portion of the sources of evolutionary variation as 'innovation', albiet with the obligitary disclaimers about teleological implications. Old analogies into the social and cognitive domains will be reexamined, especially their consequent implications of optimality, and new ones based on metaselection for innovation will be explored. To those ends it will be necessary to confront some of the combinatorial mathematics of possibilities and actualities, as well as to examine the limits of applicability of evolutionary epistemology. By way of conclusion this essay will note possible barriers to invoking innovation as a component of recent speculations on cosmological evolution.
The success of reductionist biochemistry in elucidating a mechanism for inheritance has convinced many scholars that the behaviour of creatures with central nervous systems might be similarly reducible to genetic determinants. Because of significant doubts as to the tangibility of 'atoms' of behaviour or cognition [Smith, 1993], this essay provides the counterpoint of considering how the dynamics of biological evolution might inform us as to the nature of psychological and social development.
This essay has been informed by Jeffrey S. Wicken's  discussion of the interrelationship of selection pressures and thermodynamic requirements on whole ecosystems and their component species. The succession of ecosystems must at each step employ species with their own histories of continuing viability to efficiently fill niches in the ecosystem-wide network of energy transformations. Living things must balance multiple 'goals' in order to live, grow and reproduce, and the flexibility that is required to meet those goals is similarly required over generations to survive the comings and goings of their host ecosystems.
Recent research across the kingdoms of life has revealed many examples of 'genetic engineering' in nature. J. A. Shapiro  notes that for "(t)he kind of genetic engineering that is practiced in research laboratories and biotechnology companies … (v)irtually all the methods used … employ enzymes and systems extracted from living cells". Even before Watson and Crick elucidated the structure of DNA, Barbara McClintock's work on the genetics of plants [Fedoroff 1992] had revealed the role of natural genetic recombination in introducing novel functions. Bacteria regularly exchange genetic material between more or less related lines using plasmids and bacteriophages, while what are similarly and conventionally described as 'parasitic' retroviruses can play a similar role in copying genetic elements between unrelated species. Some protozoans have been shown to completely reorganise their genetic material in the space of a single cell division cycle, while many plants and invertebrates are able to incorporate large scale reorganisation into their germ line so as to provide a reasonable chance of establishing a viable founder population. While the possibilities for such rearrangements are clearly more restricted in the warm blooded invertebrates (where they may be compensated by reproduction of acquired behaviours through parenting), the mammalian immune system actively employs genetic recombination to provide an unrestricted variety of highly specific antigen binding sites.
Shapiro  concludes that "the thinking of evolutionary theorists about conceivable mechanisms of genetic variation should be freed from restrictions imposed when knowledge of genetic mechanisms and DNA biochemistry was still rudimentary." This essay contends that the active application of a broad range of genetic engineering techniques by life forms on (largely their own) genomes is the dominant source of genetic variation and that is therefore much more appropriate to describe that variation as 'innovation' rather than as random mutation. Given that mechanisms for genetic innovation exist, it is now possible to show that they could have been expected to arise over the course of evolutionary history.
While some progress is being made on identifying traces of pre-Cambrian life, we must rely overwhelmingly on the progeny of the Cambrian 'explosion' of animal phyla to study patterns in the record of speciation and extinction. All chordates, arthropods, molluscs, echinoderms and other advanced animals alive today have a continuing lineage of ancestors which were viable in their time back to a common ancestor which successfully differentiated its early embryo into three rather than two distinct layers of cells. The third layer may or may not have been a factor in the viability of what for some period must have been a single species in a world of less flexible creatures and its potential impact on future evolution could not have been anticipated until early instability of development pathways for mesodermal tissue allowed rapid 'experimentation' with more and more complex body plans.
It is widely recognised that there have been five episodes of 'mass' extinction since the Cambrian, with these step changes in the fossil record being the markers which separate the major geological periods. There is increasing evidence that there have been many lesser 'extinction events' scattered through the 550 million years, to the point that it has been contended that most extinctions of species have been due to such events. It appears certain that a range of causes have initiated such events from the impacts of extraterrestrial bodies to climatic instabilities produced by changing continental allignments. Extinction events involved the collapse of various ecosystems and generally eliminated those species which were primarily dependent on the collapsed ecosystems, leaving a few wider ranging or behavioually flexible generalists to inherit the earth.
Within a relatively stable ecosystem, selection works very differently. Species which establish a clearly differentiated niche will prosper increasingly as their exploitation of that niche becomes more efficient. What is good in the good times becomes a disaster when that ecosystem collapses due to 'selection criteria' which bear no relationship to those applying in the stable ecosystem. It may be the most fortuitous coincidence of all that extinction events have been sufficiently frequent for enough orders of generalists to survive each one to be able establish not just viable replacement ecosystems but, after up to ten million years recovering from the big crashes, to produce both species and ecosystems which were more complex than any that had preceded them.
On land, where boundaries between ecosystems are most clearly marked, the most pervasive species exhibit one or another specific capacity for flexibile development: (i) the flowering plants exhibiting morphological flexibility, (ii) the social insects exhibiting organisational flexibility, and (iii) the warm blooded vertebrates exhibiting behavioural flexibility. This observation produces serious questions for the 'structure cause of function' argument of O'Grady and Brooks (1988). If a species is sufficiently flexible to start exploiting some new opportunity or niche, then a sub-population which does so will be exposed to new selection pressures which may lead over time to speciation. It is my contention that the exploitation of new opportunities by flexible species is the primary source of the speciation needed for new ecosystems to become established following extinction events.
There is overwhelming concensus that the primary mechanisms of evolution can be most generally described as 'variation' and 'selection'. Still at issue is the relative influence of the two mechanisms in constraining the form and function of evolution's products, and the 'directedness' or not of the variation process.
The original neo-Darwinian synthesis leaves it to relatively rare random mutations to provide the source of variation, with the cumulative effect of persistent selection pressures assumed to drive species towards maximum 'fitness'. The rejection of Lysenko's proposed heritability of acquired characteristics has proved reasonably accurate for animal species because of the lack of a sensible mechanism for transfering an acquired characteristic back to the germ line, save and accept the reasonably rare episodes of transcribing retroviral RNA into germ line DNA. There is however a growing body of evidence for routine inheritance of acquired characteristics by bacteria. And since the advent of active parenting by animals with nervous systems, there has been an increasing ability to transfer acquired behaviours from one generation to the next—the predisposition for which can be eventually incorporated into the germ line through the changes to selection pressure following the behavioural innovation (as discussed in the previous section).
While selection clearly favours more efficient occupants of ecological niches, there are many reasons to believe that the products of selection over geological time still do not approach anything like the 'optimal' performance that an engineer would specify. For any real niche in any persistent ecosystem, there is enough local variation in selection pressure to seriously smudge the fitness landscape—a variability which is provided by a (sexual) species's gene pool having unlimited opportunities for recombination. While clearly deletorious and advantageous genes do change in frequency with time, livestock breeders wel know the difficulty of recapturing the particular combinations that produce the occassional 'champion' as determined by some or other narrow measure. The combinatorial possibilities in a species's gene pool (of order 2n) clearly exceed the actual individuals which are subject to selection in a countable population over countable generations (of order n2). And general systems theory tells us that an optimal system cannot contain any optimal components—with the fact of life on this planet being systems of systems of systems ad nauseum rendering meaningless the very concept of local optima.
Genetic recombination and behavioural flexibility together provide the equipment with which species fill niches opened up by the occasional collapse of ecosystems, so that over sufficient cycles of prosperity and extinction, those life forms with the greatest capacity to innovate have come to dominate. Modern species exhibit great variability within their respective gene pools as well as retaining mechansims for transposing genes with a reasonable likelihood of changing the regulation regime applying to the expression of certain sets of proteins so that, for example, the increased flexibility of juvenile behaviour is often tested through paedomorphisis. The coevolution of flowering plants with pollen and seed dispersing animals has meant that even a kingdom which exhibits nothing that can sensibly be called 'behaviour' has been able to greatly increase its species' organisational complexity as a result of the behavioural innovation of other species. It is time we stopped thinking about biological evolution as a process which achieves wonders through the powers of selection acting on random variations and instead realised that the history of life, at least since the Cambrian, has 'metaselected' for species with a capacity to produce and sustain innovation both behaviourally and genetically.
Such innovations necessarily accumulate, the remotest possibility of truly reversing them disappearing as soon as they become the foundation on which further innovations are built. Things that might have been, like technological dinosaurs, are closed off by extinction events, just as those with the flexibility to survive and fill new niches find opportunities for innovation that are within genetic and/or behavioural reach of their established nature.
The influence of the theory of evolution on thinking in other areas is already profound. Epistemology has been proposed to be limited to those things which humans have evolved the capacity to know. The metaphorical correspondence between evolution in the biological and social domains has been considered at length and, more recently, similar analogies have been applied to some physical systems and human cognition.
The emerging sciences of complexity on one hand and the more established studies of general systems and cybernetics on the other appear to me to be converging on a shared understanding of a general mechanism governing the emergence of new levels of organisation in all domains (near the 'edge of chaos') of which the origin of biological species may come to be seen as one particularly rich exemplar.
The particular concern of this essay is to identify active mechanisms for innovation and corresponding selection constraints applying in other domains. It is relatively easy to do this in social and cognitive arenas, to the degree that I would argue that it is now useful to reverse the metaphor and begin looking for biological phenomena which are analogous to phenomena identified in the more dynamic social domain. However, when the analogy is stretched to cosmological questions [see Gribbin 1993] we can take a less cluttered look at the relationship between historical viability and innovation as well as asking some serious questions about the very applicability of selection.
Having seen that an active process of innovation has been essential to the growth of organisational complexity through biological evolution, we should consider afresh the evidence for similar contributions from processes of innovation in analogous domains. Having identified that wider evidence for the essential role of innovation, I will proceed to suggest some implications for the sociopolitical agenda.
We have seen that behaviour represents the dominant form of heritable innovation in the ward blooded vertebrates in particular. These creatures are hatched or born with a very plastic capacity for developing behavioural repertoires—with the exceptions (that is 'hard wired' or instinctive behaviour such as 'imprinting') having been the subject to disproportionate notice because of their reproducibility and sometimes nonsensical consequences. The young of such creatures have a strong disposition to innovative behaviour in the form of play and exploration which is constrained through active parenting and in many cases social pressures, ultimately passing on the heritage of a behavioural repertoire which has proved viable for the species. These innovations are only really innovative for the individual concerned, being neither pre-programmed nor passively acquired through some process of transference. Those that are genuinely innovative at a species level will frequently have fatal consequences for the young innovator, however the few behavioural innovations at a species level which do prove viable appear to be the main source of evolutionary 'progress' for such species (and their symbiotic beneficiaries). In some mammalian species the extra behavioural freedom open to adolescent males nurtures innovation.
Inappropriate metaphors from other domains have long produced wild theories to guide would be social architects, however the fact of evolutionary mechanisms producing increased organisation in biological and social domains cannot be denied and so should be the source for more useful metaphors. Economic 'rationalism' was inappropriately based on the near equilibrium physics of closed systems, more recently extended by the false neo-Darwinian goal of optimality.
Despite such effective baselessness, the fact that market economies have proved more resilient than centrally controlled economies is justification to look for a new theoretical understanding. While planned economies may be 'jump started' by strong propagation of some or other innovative doctrine, they do not retain the ability of market economies to progressively incorporate a stream of innovations, and so become less and less able to cope with changes in their environment.
Science and technology provide exemplars of the power and form of innovation in a social context. While the purported object of science and academia is to discover truths, they are better understood as enterprises for the purpose of inventing more and more useful representations of the experiential world which can be propogated in language and, more recently, in the simulacra. Such knowledge representations build upon others which have already proved viable, producing an ever expanding edifice. At least technology pursues innovation for its own sake without apology. In so doing it continues to transform the lifeworld as well as providing the most visible model of an evolutionary process based openly on innovation. At every opportunity, many new technological innovations are proposed, several coming into use within a short period of time but most often just two incompatible approaches remaining viable for long enough for them to become the bases on which further innovations are developed.
Technical innovation is now widely seen as something to be encouraged albiet that the mechanisms claimed to actually provide such encouragement are generally inconsequential. However, in other areas of social organisation and social behaviour, innovation is usually treated as something to be resisted and obstructed at every turn, not the least of which to protect fragile orthodoxies of the 'politically correct'. Those lacking the courage to allow social and behavioural innovation to contribute to the solution of pressing social problems have been able to rely on a historical succession of authorities—from God to optimality—to support the status quo. However recognition of the role of innovation in producing more and more complex and thus responsive organisations at the 'edge of chaos' may finally enable us to contemplate a society more in keeping with the range human aspirations than any that economic rationalism or its failed former rivals could ever bring.
One of the greatest obstacles to a wider recognition the power of a cummulative history of viable innovations is common confusion over the ability of design processes to test possibilities. Without any 'feel' for orders of magnitude beyond their everyday experience, the general public largely assume that given enough time and enough monkeys tapping away on typewriters that the works of Shakespeare would be generated. They just have no feel for the fact that the combinatorial number of possibilities for the arrangement of even a modest set of simple alternatives, such as those in a biological species's gene pool, is incomporably larger than the number of anythings this side of the event horizon. While they are still far from impacting even the educated public, the increased success of genetic algorithms and neural nets in solving problems in a way that no engineer would consider may help spread the awareness. Beyond that, the open question is how to introduce the foundations of a (second order) cybernetic understanding sufficiently early in the education process that the benefits of plurality and innovation will become accepted as superior to those of prescription and control.
The metaphor of evolution, with its implications of ongoing change of form, has been widely taken up in the physical sciences to describe systems from 'living' planets to whole galaxies. Taking a further step in removing humanity from centre stage, John Gribbin  sees our physical universe as one reproductive act of a successful line of near singularities  which must have evolved analogously to life on earth. Since the abandonment of an early model suggesting that a passing star must have sucked the material that formed our solar system out of the sun, and leaving aside the currently accepted explanation for the origin of the moon, there is no sign in these various processes of physical evolution of the main icons of biological evolution—not sexual combination, nor coding of heritable information, nor the selection pressures of resource competition.
Looking at evolution in the physical domain removes some obstacles to a clear understanding of the essential basis of any evolutionary process in its history. Where there is neither coded information nor competitive selection, it is a little more obvious that those organisations which are most likely to occur will be those which occur most frequently, particularly those which innovate viably near the edge of chaos. When a succession of generations follow similar paths, differences in frequency will grow exponentially. So, Collier's  assertion "that evolution is possible with no environmental selection" may prove essential if we are to treat seriously the extension of general evolution to cosmology. I know that we need an evolutionary process on that scale to produce sufficient complexity to match observations in this cosmos and to support evolution of the further complexities of biology, mind and society. I am also well aware of how much resistance there will be to such a notion, to say nothing of the difficulty in even conceiving of how we ever got from a literal nothing—no space, no time, no quantum fields to have quantum fluctuations—to the complexly organised patterns our planets, lifeforms and technologies faintly etch on the rigidly expanding manifold of our cosmos.
A G Cairns-Smith, 1982. Genetic Takeover and the Mineral Origins of Life, Cambridge UK: Cambridge University Press.
Collier, John, 1988. "The Dynamics of Biological Order". In Entropy, Information, and Evolution: New Perspectives on Physical and Biological Evolution, Bruce H. Weber, David J. Depew, and James D. Smith (eds.), Cambridge MA: MIT Press, pp. 227-242.
Collier, John, 1993. Replying to "Evolution as Entropy", as redirected to bionet.info-theory. December 1993.
Fedoroff, Nina, and David Botstein (eds.), 1992. The Dynamic Genome: Barbara McClintock's Ideas in the Century of Genetics, Cold Spring Harbor Laboratory Press.
Gribbin, John, 1993. In The Beginning.
O'Grady, Richard T., and Daniel R. Brooks, 1988. "Telology and Biology". In Entropy, Information, and Evolution: New Perspectives on Physical and Biological Evolution, Bruce H. Weber, David J. Depew, and James D. Smith (eds.), Cambridge MA: MIT Press, pp. 285-316.
Shapiro, J. A., 1992. "Natural Genetic Engineering in Evolution". In Transposable Elements and Evolution, Special Issue of Genetica Vol. 86, Nos. 1-3, pp. 99-111.
Smith, Tony, 1993. "Culture, Metabolism and Atomicity" posted to PRNCYB-L. October 1993.
Wicken, Jeffrey S., 1987. Evolution, Thermodynamics, and Information: Extending the Darwinian Paradigm, New York: Oxford University Press.