My goal in 2012 is to write at least one paper and try to get it published. The paper I have in mind is inspired by three men, and their corresponding books. These are Friedrich Hayek and The Constitution of Liberty, Thomas Sowell and Knowledge and Decisions, and Everett Rogers and Diffusion of Innovations. I want to put the pieces together in order to make a single, solid argument, but I suspect I’m going to need a few more pieces before I can get there.
F. A. Hayek: Trial and Error and Local Knowledge
At any stage of this process there will always be many things we already know how to produce but which are still too expensive to provide for more than a few. And at an early stage they can be made only through an outlay of resources equal to many times the share of total income that, with an approximately equal distribution, would go to the few who could benefit from them. At first, a new good is commonly “the caprice of the chosen few before it becomes a public need and forms part of the necessities of life. For the luxuries of today are the necessities of tomorrow.” Furthermore, the new things will often become available to the greater part of the people only because for some time they have been the luxuries of the few.
-Friedrich Hayek, The Constitution of Liberty
Hayek argued that everything in human society–from technology to words to ideas to norms–begins its life as something developed and adopted by a small subset of the population. Some tiny fraction of these end up gaining mainstream adoption.
When I read The Constitution of Liberty two years ago, I became enamored by this very simple framework. It seemed an elegant explanation for how cultures evolve over time, through a process of rote trial and error.
On the other hand, I found the fact that Hayek didn’t elaborate on the process any further to be frustrating. If I had my way, I would throw out every last section of that book except the bits on cultural evolution, and have had him make up the other 400 some pages by digging deeper into this concept.
What Hayek is known for more widely is his work on local knowledge. In particular, “The Use of Knowledge in Society” discusses how the price system makes it possible for people to act on their specific knowledge of time and place without needing to get the much more difficult to acquire big-picture knowledge. Speaking of a hypothetical man on the spot, he wrote:
There is hardly anything that happens anywhere in the world that might not have an effect on the decision he ought to make. But he need not know of these events as such, nor of all their effects. It does not matter for him why at the particular moment more screws of one size than of another are wanted, why paper bags are more readily available than canvas bags, or why skilled labor, or particular machine tools, have for the moment become more difficult to obtain. All that is significant for him is how much more or less difficult to procure they have become compared with other things with which he is also concerned, or how much more or less urgently wanted are the alternative things he produces or uses. It is always a question of the relative importance of the particular things with which he is concerned, and the causes which alter their relative importance are of no interest to him beyond the effect on those concrete things of his own environment.
Hayek’s entire worldview was built around the idea of complex human systems which required more knowledge than any one individual within them could possibly have, something that Leonard Read captured more poetically in “I, Pencil“. The process of cultural evolution involved individuals and small groups trying out something new, which is observed by others who decide whether or not that new thing fits in with the particulars of their own circumstances, needs, and taste. In short, it doesn’t require much knowledge to come up with something new, and then an incremental amount of local knowledge is brought to bear as more individuals get exposed to that new thing.
But, as I said, he didn’t develop this system in any real detail.
Thomas Sowell: Knowledge Systems
The unifying theme of Knowledge and Decisions is that the specific mechanics of decision-making processes and institutions determine what kinds of knowledge can be brought to bear and with what effectiveness. In a world where people are preoccupied with arguing about what decision should be made on a sweeping range of issues, this book argues that the most fundamental question is not what decision to make but who is to make it–through what processes and under what incentives and constraints, and with what feedback mechanisms to correct the decision if it proves to be wrong.
-Thomas Sowell, Knowledge and Decisions
Sowell begins Knowledge and Decisions by explicitly recognizing his intellectual debt to Hayek in general and “The Use of Knowledge in Society” in particular. Yet in the book he goes far beyond any level of detail that Hayek provided on the subject, at least that I am aware of.
One of the crucial components of the book is the emphasis on feedback mechanisms.
[F]eedback mechanisms are crucial in a world where no given individual or manageably-sized group is likely to have sufficient knowledge to be consistently right the first time in their decisions. These feedback mechanisms must convey not only information but also incentives to act on that information, whether these incentives are provided by prices, love, fear, moral codes, or other factors which cause people to act in the interest of other people.
Clearly, feedback mechanisms must play a huge role in Hayek’s process of social trial and error. Feedback mechanisms are what determine what is considered “error” and force people to change course. As Sowell explains, they take many forms:
A minimal amount of information–the whimpering of a baby, for example–may be very effective in setting off a parental search for a cause, perhaps involving medical experts before it is over. On the other hand, a lucidly articulated set of complaints may be ignored by a dictator, and even armed uprisings against his policies crushed without any modification of those policies. The social use of knowledge is not primarily an intellectual process, or a baby’s whimpers could not be more effective than a well-articulated political statement.
He added “[f]eedback which can be safely ignored by decision makers is not socially effective knowledge.”
So discerning what outcomes we should expect from the various forms of social trial and error requires identifying the relevant feedback mechanisms. The feedback that potential new words faced takes a very different form than the feedback a new product on the market faces, or a publicly funded project.
The particulars of these feedback mechanisms, along with the incentives and institutional context, determine “what kinds of knowledge can be brought to bear and with what effectiveness” in each given case.
In many ways, Knowledge and Decisions is just good old-fashioned economics–it deals with incentives, with inherent trade-offs, and with scarcity. But it is a particularly Hayekian take on economics, with its focus on the scarcity of knowledge in particular and the role of very localized, difficult to communicate knowledge.
I don’t think Sowell gets nearly enough credit for this work among economists generally or even among Hayekians.
Everett Rogers: Curator of His Field
This book reflects a more critical stance than its original ancestor. During the past forty years or so, diffusion research has grown to be widely recognized, applied, and admired, but it has also been subjected to constructive and destructive criticism. This criticism is due in large part to the stereotyped and limited ways in which many diffusion scholars have defined the scope and method of their field of study. Once diffusion researchers formed an “invisible college” (defined as an informal network of researchers who form around an intellectual paradigm to study a common topic), they began to limit unnecessarily the ways in which they went about studying the diffusion of innovations. Such standardization of approaches constrains the intellectual progress of diffusion research.
Everett Rogers, Diffusion of Innovations, 5th Edition
After I read Constitution of Liberty, I realized that there was probably a literature behind the kind of phenomena that Hayek was talking about. The term “early adopter”, which has become part of the mainstream lexicon, must have come from somewhere. Hayek was unfortunately of little help; he cited old theorists like Gabriel Tarde. While the diffusion literature owed a certain intellectual debt to Tarde, he was writing nearly half a century before the modern field emerged.
I eventually happened upon Diffusion of Innovations, Everett Rogers’ book, the various editions of which basically bookend the entire history of the field in his lifetime. Which is quite helpful, because it began in his lifetime–and the first edition of the book was instrumental in its formation.
Where Hayek and Sowell’s works are within the confines of high theory, Diffusion of Innovations is a thoroughly empirical book, at times painstakingly so. There is not a single concept that Rogers introduces, no matter how simple, which he does not illustrate by summarizing a study or studies which involve an application of that concept.
Rogers helped formalize many of those concepts himself with the first edition of the book, published in 1962, when the literature was pretty sparse and dominated by rural sociologists. Since then, it has expanded across disciplines and in volume of published works. As a result, in the last edition of the book, published only a year before he died, there were many aspects of the diffusion process that had been solidly demonstrated by decades of work.
The books always served as a tool for both introducing the field to those unfamiliar with it, and attempting to steer future work. In the final edition, Rogers highlights not only what the literature has managed to illuminate, but its shortcomings. In short, the book has just about everything you would want if you were attempting to get a sense for what work has been done and what has been neglected.
There are aspects of the diffusion literature which are quite Hayekian. In particular, the emphasis on uncertainty and discovery processes.
One kind of uncertainty is generated by an innovation, defined as an idea, practice, or object that is perceived as new by an individual or another unit of adoption. An innovation presents an individual or an organization with a new alternative or alternatives, as well as new means of solving problems. However, the probability that the new idea is superior to previous practice is not initially known with certainty by individual problem solvers. Thus, individuals are motivated to seek further information about the innovation in order to cope with the uncertainty that it creates.
The various mechanisms which Rogers describes which individuals employ to reduce uncertainty–trying the innovation on a partial basis, or observing how it goes for peers who have adopted the innovation, or measuring the innovation against existing norms, to name a few–can be seen as clear cut cases of economizing on information.
In many ways the diffusion model that Rogers lays out is the detailed system that I wanted Hayek to develop. Rogers discusses so many specific aspects of the process; such as the role of heterogeneity and homogeneity, people who are more cosmopolitan or more localite, the different categories of adopters–including the familiar early adopters–and on and on. Rogers concisely describes and categorizes the various feedback mechanisms against adoption in the system.
On the other hand, the beginning of the process–the actual generation of the innovation–is where the literature is by far the weakest. Rogers cites several who have criticized it for this, and agrees that it is a problem. He points out several attempts that have been made to address this problem, but it’s clear that not nearly as much work has been done nor are the results as solid.
Part of the problem is the historical origins of the field–the diffusion literature began with rural sociology, where innovations were developed in universities who then peddled their wares to American farmers. The single most influential study dealt with the diffusion of hybrid corn, which seemed very clearly to be a quantifiable improvement over its alternatives. As such, many diffusion studies have the perspective of assuming that an innovation should diffuse, that there is some problem with the people who reject rather than adopt.
How did the pro-innovation bias become part of diffusion research? One reason is historical: hybrid corn was very profitable for each of the Iowa farmers in the Ryan and Gross (1943) study. Most other innovations that have been studied do not have this extremely high degree of relative advantage. Many individuals, for their own good, should not adopt many of the innovations that are diffused to them. Perhaps if the field of diffusion research had not begun with a highly profitable agricultural innovation in the 1940s, the pro-innovation bias would have been avoided or at least recognized and dealt with properly.
Moreover, the outline of what he believes is the process by which innovations are generated is a very directed, top-down process. It involves “change agents” that are consciously attempting to solve problems and diffuse some innovations. I’m not arguing against the existence of such agents–they are obviously an extensive part of society, from medical researchers seeking a cure for cancer and pharmaceutical companies attempting to get their drugs mainstream adoption, to Apple coming up with a completely different kind of smartphone and tablet and bringing them to market.
But the change agents, as Rogers and the diffusion literature envision them, are only a part of Hayek’s story of social trial and error. Consider language–new words and phrases emerge all the time and diffuse through a process which I am certain is identical to the one Rogers describes. On the other hand, I highly doubt that there are “change agents” who developed these new words and phrases in a lab somewhere and then promoted them. I think the process is far more organic.
Rogers also discusses the role of norms in terms of how they hinder or help the diffusion of an innovation, but left unsaid I think is that those norms are themselves undoubtedly the product of a previous diffusion. In Hayek and Sowell’s framework, traditions and existing norms emerged in response to trade-offs that needed to be made throughout a culture’s history. As Edmund Burke put it succinctly in Reflections on the Revolution in France:
We are afraid to put men to live and trade each on his own private stock of reason; because we suspect that this stock in each man is small, and that the individuals would do better to avail themselves of the general bank and capital of nations, and of ages.
The trial and error process that Hayek envisioned built up that “general bank and capital of nations, and of ages” as societies developed increasingly effective ways to manage their trade-offs.
Rogers does touch on this point of view from a couple of angles. First, he describes the work of Stephen Lansing in uncovering the astonishing effectiveness of the local knowledge contained in the religious hierarchy of Bali, as he described in his book Priests and Programmers. This was a case where the seemingly beneficial innovations of the Green Revolution proved inferior to what seemed like mere superstitious practice.
The Balinese ecological system is so complex because the Jero Gde must seek an optimum balance of various competing forces. If all subaks were planted at the same time, pests would be reduced; however, water supplies would be inadequate due to peaks in demand. On the other hand, if all subaks staggered their rice-planting schedule in a completely random manner, the water demand would be spread out. The water supply would be utilized efficiently, but the pests would flourish and wipe out the rice crop. So the Jero Gde must seek an optimal balance between pest control and water conservation, depending on the amount of rainfall flowing into the crater lake, the levels of the different pest populations in various subaks, and so forth.
When the Green Revolution innovations were introduced to the region, crop yields dropped, rather than increased. This intrigued Lansing.
In the late 1980s, Lansing, with the help of an ecological biologist, designed a computer simulation to calculate the effect on rice yields in each subak of (1) rainfall, (2) planting schedules, and (3) pest proliferation. He called his simulation model “The Goddess and the Computer.” Then he traveled with a Macintosh computer and the simulation model from his U.S. university campus to the Balinese high priest at the temple on the crater lake. The Jero Gde enthusiastically tried out various scenarios on the computer, concluding that the highest rice yields closely resembled the ecological strategies followed by the Balinese rice farmers for the past eight hundred years.
Clearly, Balinese society had arrived at this optimal solution through some process. But Rogers does not delve too deeply into this.
Rogers also acknowledges that the literature may have focused too exclusively on more centralized processes.
In recent decades, the author gradually became aware of diffusion systems that did not operate at all like centralized diffusion systems. Instead of coming out of formal R&D systems, innovations often bubbled up from the operational levels of a system, with the inventing done by certain lead users. Then the new ideas spread horizontally via peer networks, with a high degree of re-invention occurring as the innovations are modified by users to fit their particular conditions. Such decentralized diffusion systems are usually not managed by technical experts. Instead, decision making in the diffusion system is widely shared, with adopters making many decisions. In many cases, adopters served as their own change agents in diffusing their innovations to others.
Though recognizing that such processes exist, it’s clear that the work that has been done on this is much thinner than the more traditional, change agent based research.
Questions That Remain
As I said, all three of these pieces have some holes in them, and those holes aren’t necessarily filled just by putting all of them together.
The next logical step would probably be to seek out more material like Rogers’, where a lot of work has been done and concrete conclusions can be drawn. Any work on how new words and phrases emerge and proliferate would probably be a good start.
Online communities also have many customs, such as hashtags on Twitter and the hat tip among bloggers. The advantage to customs like this is that they leave behind recorded evidence, unlike, say, an oral tradition. We know, for instance, when hashtags first became popularized among Twitter users–it is documented. A great deal of work is being done by communications scholars on subjects such as these; this could also probably provide some more solid leads.
What I want to argue is that innovations are generated in a Hayekian trial and error process, and some subset of them gain mass adoption in the manner described by the diffusion of innovations literature. I want to describe the role that local knowledge plays in that process; how the feedback mechanisms and incentives shape what innovations are generated and which ones ultimately are adopted.
But there’s more research to be done before I can make a case for this thesis that is solid enough for me to be comfortable with.
