The Origins of Human Social Systems

Diffusion is the Engine

Everything that is now mainstream was once obscure. We are now largely aware of this fact, especially those of us who follow technology trends. The “early adopter” is a familiar friend, rather than merely the subject of obscure communications papers.

The study of how the obscure becomes widely adopted began in the mid-20th century in rural sociology. The origin of this field is usually credited to the research of Neal Gross and Bryce Ryan on American farmers’ adoption of hybrid corn in the 1940s. Gross and Ryan relied mostly on structured surveys to gather information on why farmers had resisted the innovation, or chosen to adopt with the particular timing that they had.

A great deal of research continued to be done along these lines, until in 1962 Everett Rogers decided to summarize the findings in the first edition of Diffusion of Innovations. The book was more than a simple account of the literature; it provided a basic theoretical framework, as well as suggestions for the direction of future research. An avid participant in the field throughout his life, Rogers continued to put out new editions of the book to bring it up to date and gently guide future research based on how it had progressed between editions. The fifth and final edition came out in 2003 and included some preliminary research on diffusion patters on the Internet. Rogers died not long after, so there will be no sixth editions summarizing studies on diffusion patterns on Twitter, Facebook, or app stores.

Diffusion of Innovations provides nothing so elegant as a theoretical framework; instead, there is a set of stylized observations, summarized from a large body of research which has developed over more than sixty years. This is not to say that it is without theory—just not the kind of theory you might find in The Wealth of Nations or Civilization and its Discontents. Nothing grand—just a basic framework with a number of theoretical concepts baked in, taken mostly from preexisting works.

One useful concept Rogers and his colleagues honed down is what exactly an “innovation”, the unit of diffusion, is: “an idea, practice, or object that is perceived as new by an individual or other unit of adoption”. With this definition, we don’t need to quibble about what is “really” an innovation or “really” new; so long as it is “perceived as new” by whatever “unit of adoption” we are considering. This is the only manner that the term “innovation” will be used for the rest of this piece.

The diffusion of innovations literature suffered from a short-sightedness due to its focus on specific innovations that were believed by the researchers to be objective improvements over pre-existing alternatives, or in any case an alternative world in which they do not diffuse. Rogers was aware of this, and took his colleagues and his own work to task for what he called a “pro-innovation bias”. Even still, they managed to develop a very informative framework from which to view diffusion.

The hallmark of this literature is the standard s-shaped diffusion curve. When an innovation appears among an observed population, at first it diffuses very slowly among a very small subset of the population. Rogers doesn’t quite spell it out, but this is where a vast supermajority of innovations die. An enormous amount of energy and money has been spent trying to figure out what it is that allows some tiny fraction of innovations to survive beyond this crucial stage, but our knowledge is still very partial. At best, we can tell what increases or decreases the relative probabilities of survival, though what the actual probabilities are we cannot say.

800px-Diffusion_of_ideas.svg

Nevertheless, some innovations do survive. Once they have moved beyond the people categorized as “the innovators” and “the early adopters”, we hit “the early majority”, and the diffusion process takes off like a rocket. As Rogers put it:

The part of the diffusion curve from about 10 percent adoption to 20 percent adoption is the heart of the diffusion process. After that point, it is often impossible to stop the further diffusion of a new idea, even if one wished to do so.

One of the things that increases the probability that one person will adopt an innovation is if it has already been adopted by other people who are a lot like them that they know. The underlying assumption, supported by a variety of research, is that something like 60 percent of every population is homogenous, and innovations spread rapidly once they reach this group. There is a prior group of about 10-20 percent of the population that are quicker to adopt and are usually wealthier or higher status on some margin than the majority. Finally, the remaining population, called (pejoratively—there’s that pro-innovation bias again) “laggards”, only adopt after the majority, and only do so slowly. They tend to be poorer, or older, or lower status on some margin from the majority. The relative percentages of the population made up by each category may vary but there is a stable observed phenomena where there’s the first slow group, the second, biggest fast group which constitutes a majority, and the last slow group.

I believe that this model can be extended far beyond where the researchers have taken it. But to do so, several important questions will have to be answered. While there exists an enormous body of studies on successful and failed diffusions, there are not many studies on what happens after a successful diffusion—the average lifespan of successfully diffused innovations, so to speak. There is some research on how prior diffused innovations lay the foundation for future ones, but I would prefer to see a lot more on this.

I think that the diffusion model describes the dynamics of how norms, traditions, and every human social system emerges. It begins with a set of discrete innovations that build on top of each other until you get path dependence, feedback loops, and ultimately, the large, complex arrangements of the modern world.

But more on the big picture a little further down.

First, let’s look at the small picture.

Networks are the Fabric

Paul Adams’ Grouped provides an excellent summary of the literature on the structure of human networks. And here we mean not “networks” in the technological sense in which the word has come to be used, but networks in the older sense of connections among people. Networks in this sense have existed for as long as human beings have been social animals; which is to say, for as long as they have been human beings.

from Grouped
from Grouped

In the “strong ties, weak ties” literature, we are described as having up to 15 strong ties, and as many as 500 weak ties that we can actually remember something about. Of those, Robin Dunbar has famously found that we are only capable of maintaining stable relationships with 150 people total, including our strong ties. Our weak ties are made up of people who come in and out of that 150 over time, and some never get beyond that threshold.

Of our strong ties, 80 percent of our communications happen with our closest set of up to five connections. Our strong ties exert enormous influence on us, and these five connections are by far the most influential in our lives. A lot of what we like to think of as discrete, individual tastes are really things that people only like because the people they know like them. A great deal—perhaps the majority—of preferences owe their origin to this groupish tendency rather than to some innate internal ranking what someone wants.

All of our friends, even our very closest ones, have their own circles of connections that overlap to greater or lesser extent with ours. The further out into their weaker ties you go, the more unique connections they have that you do not have. The extended connections of your closest connections alone make up many, many thousands of people. These semi-overlapping circles comprise a network that encompasses basically all of mankind, with a few outliers. This is what is meant when it says that everyone can be connected to anyone else within six degrees of separation; and that number has decreased with the Internet and social networks like Facebook.

However, even three degrees is actually a vast social distance. Our personal influence on the group three degrees removed from us is very minimal, compared to the influence we have on our closest five friends, and the influence that they have on us. However, information and influence does come from outside of this inner circle. After all, the people in your inner circle have people in their inner circle that aren’t in yours, who have people in their inner circle that aren’t in your friends’, and so on. Through this network of close connections, information, preferences, norms, slang, and just about every sort of innovation diffuse.

These small networks are the fabric that every social system is stitched from. If one of your five closest connections adopts a certain norm, the odds a very high that you will too—higher, in any case, than if it was just one of your 15 closest connections, or one of your 50 closest, and so on. If one of your weak ties adopts an innovation (especially the 135 or so that you’re able to keep a persistent connection with), it does increase the odds that you will adopt it as well—just not by very much.

People we think of as highly influential have an enormous number of weak ties, but no more strong ties than anyone else. If one of these people adopts an innovation, there’s a small probability that their weak ties will. But that small probability still constitutes an increase from before they had adopted it—and every so often something will take off among their weak ties.

Nevertheless, Paul Adams is firm on the point that the overwhelming majority of influence occurs within the loci of these small networks; regular people influencing other regular people who constitute their closest connections; and those people influencing yet other people who constitute some of their closest connections. We influence, and are influenced in subtle, almost invisible ways, by our closest connections. Agenda setting, and other top-down models of influence, are bogus.

Diffusions Accumulate into Traditions

We can think of these small networks as going through an endless trial and error process, in which the vast supermajority of innovations are rejected as errors. Of those innovations that do diffuse more widely, the overwhelming majority do not survive longer than a year (we call these “fashions”). This is clear from the logic of the Lindy Effect; first formulated by Benoit Mandelbrot and lately popularized by Nassim Taleb. The Lindy Effect states that every technology (understood in the broadest possible sense) is on average halfway into its lifespan; meaning that new innovations are almost always a flash in the pan.

The Lindy Effect also makes it clear that any innovation that has lasted a long time, will last a long time yet. So some tiny fraction of initial innovations diffuse, and a small fraction of those innovations survive their first year post-diffusion. Over time, these survivors accumulate to form the basis of our norms and traditions.

Michael Oakeshott described specific traditions as voices in the conversation of mankind, but I think this provides too great a sense of uniformity within each tradition. To me, each tradition is more like a conversation in itself, that has been going on for a very long time, and whose participants fluctuate within a given time span and across generations.

These conversations are the product of an inconceivably large number of previous diffusions. Over time, traditions develop their own vocabularies, styles of discourse, and patterns of behavior. These emerge as the result of specific innovations—specific terms, specific discourses that exhibit particular characteristics, specifics behaviors—diffusing, each diffusion increasing the probability that future successful innovations will be related to those that have already been adopted.

The conversation metaphor should not imply that traditions are primarily composed of words and articulated concepts. Though each tradition does have a set of articulated stories, these are a small fraction of the whole of a tradition, which is primarily made up of unarticulated, tacit, practical knowledge. We know when we have made a faux paus within the context of a particular tradition even if no one explicitly tells us so, because our fellow participants have a wide variety of ways of signaling this to us. More to the point, we are wired to keep an eye out for signs that we have breached the etiquette of the circumstances, especially when we are very young.

Where one conversation begins and another ends is not quite clear; there is the Christian tradition but then there is also the Protestant tradition, and the Calvinist tradition, and the American Presbyterian tradition. In some ways these traditions represent subsets of a larger whole, but in other ways they are divergent branches from a common origin.

Each conversation is maintained through an apprenticeship-like passing on of the practical knowledge required to continue it to members of the latest generation. Each new initiate takes from the store of practical knowledge and stories that grew from the conversation that had occurred before their birth, and each puts something back into it—though most of these contributions end up being forgotten by time entirely. Moreover, what each individual believes to be worthy of putting back into the conversation they may think they arrived at of their own volition, but in reality they were guided there by the way in which their perceptions were framed by the tradition itself. The most important part of making any decision is what information we choose to filter out or focus on as significant, and this is precisely the role played by the context that any tradition provides us. This is the reason why parallel innovations frequently occur not only in the math and sciences, but in engineering and even in art. Human talent is funneled down the relatively narrow corridors provided by the traditions they are embedded within.

Stories, practices, norms, fashions, lifestyles, and all forms of human behavior diffuse within and across these conversations. Each innovation has a non-zero probability of spreading within a given conversation. Different factors increase or decrease this probability. Within current mainstream economics, a new paper is more likely to draw attention if it makes what economists consider to be a novel point, and does it within the context of the neo-classical, Samuelsonian, rational expectations model of the world. It may be a criticism of said model, but coherence to those who practice scholarship in the context of being trained in that model will increase the probability of diffusion.

In short, coherence within the framework of a particular conversation increases the probability of diffusion within that conversation, while relative difficulty to comprehend an innovation within the context provided by a conversation decreases its odds of diffusion within that conversation. For this reason, the very things that make low-probability innovations unlikely to diffuse make them the most disruptive when they do; for it is precisely because they are so far from comfortable points of reference within a tradition that they are unlikely to be adopted.

Oakeshott argued that the various conversations of human social life are entirely independent and have distinct characteristics. It is difficult to think of a version of reality where this is actually the case. The diffusion of innovations literature demonstrates that the more like one other people are, the more likely diffusions are to spread among them. The fact that I have been exposed to the conversation of economics, among others, does not change the fact that I grew up in northern Virginia, in America, in an English-speaking middle class family, and went through the Virginia public school system from kindergarten through grad school. There are a lot of people who are similar to me on some or all of these dimensions but don’t know anything about economics or many of the other conversations that I participate in. Yet there is an enormous body of traditions that we share in common.

As a part of our small networks, we all form links across the respective conversations that we participate in. For this reason, while innovations are more likely to diffuse within a conversation than across them, the larger super-conversations are large enough to bridge the conversations quite frequently. So while it has become highly apparent to me that the Paleo Diet has spread like wildfire among the libertarian community, it did not originate there, and it is a popular diet beyond the confines of this particular subset of the population.

Nassim Taleb has expressed concern that what autonomy these conversations do have is being jeopardized by globalization and by the Internet. Drawing on the logic of island biogeography, he fears that reducing this autonomy will reduce the overall diversity of conversation ecosystem. This is no trivial concern, as maintaining the diversity of these traditions requires maintaining a diverse set of practical knowledge that it is not at all clear we can ever get back if they are lost. Taleb approaches this from a risk-management perspective: we do not know what practical knowledge we will need in the future for events and changes in circumstance that we are unable to predict today. As such, more diversity is better, as it hedges our bets more effectively.

It certainly seems clear that increased interconnection has led to more highly skewed power law distributions. However, this seems to be offset, in the sense Taleb is concerned with, by an ever-lengthening long tail of conversations. So while there are more rock stars that everyone is aware of, more and more niche conversations are taking place, developing their own terminology, practices, stories, and frames of reference. In academic life, while cross-discipline work is trendier than ever, the fact remains that the long term trend has been towards ever greater specialization, branching out from older, larger traditions and forming their own niche conversations. Taleb’s concerns need to be taken seriously, but there is some reason to hope.

Within the Network and Across Traditions

We all live within the small network point of view; we see the world through our direct experience, which includes the people that are a regular part of our lives. Up to five such people dominate the majority of our social interactions, and the Internet has only empowered this further since we can now communicate with people via chat, social networks, and email during work hours.

From this perspective, the stream of innovations seems like a constant barrage. We can mostly tune out the stuff diffusing through our weak ties but the ones that make it to our strong ties, and especially our five closest connections, are very difficult for us to resist. Most of these are a flash in the pan and are gone after a brief visit to our lives. Many of them remain relatively obscure, and some of them go viral to such an extent that there seems to be no one in our lives who has not at least heard of them.

Another part of the human experience are the various Oakeshottian conversations we participate in—that is, the traditions within which we are embedded, and the communities we are a member of that operate within those traditions. Though traditions are frequently seen as static, stagnant burdens on individuals living within the modern tide of change, they are not only dynamic, but highly volatile. The volatility comes from the large number of innovations that diffuse within them every year.

Of the small fraction of innovations that are introduced to a tradition which manage to diffuse, most are quickly weeded out by feedback mechanisms built into the tradition. This feedback is the cumulative result a huge number of prior diffusions which built upon one another to develop criteria for what sorts of innovations would be considered for adoption, and what sorts of consequences would result in rejecting innovations that had previously been adopted.

More important than feedback is time, the impact of which is similar to but broader than any one feedback mechanism. Over time, circumstances change; most innovations do not survive beyond the initial circumstances that were favorable to their adoption. Over time, current participants are swapped out for new ones, and the transmission of a tradition to the next generation may not include all of the innovations that had been adopted into it by the current or previous generations.

Our networks guide our actions in subtle, nearly invisible ways, and these networks are embedded within a set of traditions. The traditions are always fluctuating with innovations diffusing within and across them, and our networks form the connective tissue across which these innovations diffuse.

This constant process of diffusion, rejection, and persistent adoption constitutes an ongoing process of trial and error through which all human social systems evolve. I am tempted to say “progress” but it is difficult to define positive advancement for a system that involves changing norms and moral rules. Even from a material standpoint there will be more dead ends than successes, and many dead ends won’t even become apparent until generations after the fact. For all we know, the changes that birthed the great discontinuity in standards of living at the onset of the Industrial Revolution will turn out to be fatal to us in the long run.

We are not guides, arbiters, or engineers of this process. Our individual contributions are small and fleeting in the face of the enormous and persistent edifice of the traditions we are a part of, and the scale of the population living and dead that participate and have participated in them. We are, in short, mostly along for the ride.

Norms and Freedom

In his latest book, Luigi Zingales asks why economists aren’t more willing to talk about what the optimal norms are for a successful economy, rather than focusing exclusively on what the optimal laws are. Over at Modeled Behavior, Adam Ozimek asks:

Is this a libertarian, conservative, or progressive idea? If you view the pressure of social norms as a way to restrict individual freedom, then this can easily be seen as progressive or conservative, depending on the behavior being restricted.

This question has a history behind it. In On Liberty, John Stuart Mill made it clear that he considered social stigma to be a form of coercion. This was especially so when it influenced who people were willing to do business with:

For a long time past, the chief mischief of the legal penalties is that they strengthen the social stigma. It is that stigma which is really effective, and so effective is it, that the profession of opinions which are under the ban of society is much less common in England, than is, in many other countries, the avowal of those which incur risk of judicial punishment. In respect to all persons but those whose pecuniary circumstances make them independent of the good will of other people, opinion, on this subject, is as efficacious as law; men might as well be imprisoned, as excluded from the means of earning their bread.

Thomas Sowell, a Hayekian, spent a fair amount of space in Vision of the Anointed criticizing Mill for his anti-stigma arguments. For Sowell and Hayek, norms are the very fabric of the social order. They come from a school of thought dating back to Edmund Burke, Adam Smith, and David Hume. While Mill shared much in common intellectually with the latter two individuals, on this subject he is much closer to Rousseau, who believed we were born free, only to be shackled by social conventions soon after.

This debate centers on different ideas of what coercion is. On Sowell’s side of the debate, there’s a fairly clear line–if you are doing something because of the explicit or implied threat of violence, you are being coerced. The threat of refusing to do business with someone is not coercion because no one is entitled to do business with anyone; the right to choose who I do business with is an inherent part of my freedom of association. The fact that I am choosing not to do business with you because you have taken some action or hold some belief that there is a social stigma against does not make it coercion, any more than if I was motivated simply by the fact that I think you are ugly or something.

How Norms Change

The ancient Greek sophist Protagoras argued that morality is something that human beings are constantly teaching to one another, similar to how we are constantly teaching each other language. The moral sense theorists, and more recently cognitive scientists and moral psychologists, have given us an idea of the mechanisms through which this co-learning occurs.

Most of the time we are taught to stick to a set of norms that has existed for a very long time. But moral change does happen.

Take the American Civil Rights Movement as an example. I do not think that its progress should be measured in the laws it managed to get enacted. Its progress should be measured in the extent to which it moved our moral framework.

Moral changes, like all social changes, start with small groups and spread in a diffusion of innovations-like process. Most such innovations never spread at all. This social trial and error form the basis of the engine of all institutional change, moral or otherwise.

As moral change follows the logic of the diffusion of innovations, we would expect successful revolutions to have the advantages predicted by that literature. The activists of the Civil Rights Movement did not just give speeches and publish books; they engaged in many forms of verbal and visual rhetoric, and took many dramatic actions, which put their perspective in the context of traditional American ideals and religious doctrine. Though their success constituted a change in the norms of the country, it was more likely precisely because they framed the change within preexisting norms.

The Limits of Individual Influence

If you think that you can affect great changes as a lone individual, you are setting yourself up for disillusionment. In all of social life, everyone is but a tiny part of a much larger whole. Even the President of the United States, and others with even greater discretionary authority, face constraints by the very nature of the systems they are working within. Individual impact varies dramatically, to be sure, but even the most exceptional individual’s influence will always be small compared to the scope of the system that is acting upon them. It is also probably reasonable to assume that it is highly unlikely you will become the Martin Luther King, Jr. of your particular moral movement.

Once we have given up on individual exceptionalism, we are left with the same tools that human beings have been using for as long as we have formed groups. You cannot hope to shape the moral compass of a nation with a single blog post, but you are influential within the group of 100 or so people you are most closely associated with, and especially the 15 or so people in your inner circle–see Paul Adams on this subject, and his book for a more thorough review of the literature.

You must also accept that this group will have as much or more influence on you as you have on them. In both how you influence and are influenced by them, your social groups are the venue for your participation in all social change, including moral change.

Participating in Change

Dan Klein once said that he felt like he shouldn’t be in GMU’s economics department, where there were plenty of people who already agreed with him, but instead should go to a more mainstream department where he could work to change minds. This is a misunderstanding of how minds are actually changed. If Klein went to such a department, he would probably just become marginalized within that community. Rather than increasing his influence, it would almost certainly reduce it.

At GMU, a community of libertarians has formed, and a culture has developed within the department. Students who go to grad school there are immersed in that culture while they are pursuing their degree. They integrate into and are influenced by that culture to varying extents. Many then take that culture with them when they move on to other things. This is not unique to GMU’s economics department–all academic departments develop a culture of some kind, which acts upon and is acted upon by the students that pass through it.

We tend to have a broadcast model of influence in our heads–we think that by writing blog posts and going on TV we will change people’s minds. But the vast majority of influence happens at the level of a community. This is true even in exceptional cases–Marginal Revolution may be an influential blog, but the economics blogosphere as a community has more impact overall on the parameters of the discussions than any one of its members. Tyler Cowen’s biggest individual impact on this discussion is as a member of a community of high visibility individuals, such as Paul Krugman and Scott Sumner.

The norms developed within the communities of which we are a part are then subject to the dynamics of the diffusion of innovations–they could gain mainstream adoption, they could remain niche, or they could hit some middle point between the two extremes. They could persist for long periods of time at whatever level they attain, or they could flame out quickly and disappear.

To the extent that you are encouraging certain norms within your community which could eventually diffuse beyond it, you are participating in the process of moral change.

Fragility and Feedback

We have been fragilizing the economy, our health, political life, education, almost everything… by suppressing randomness and volatility.  Just as  spending a month in bed (preferably with an unabridged version of War and Peace and access to The Sopranos’  entire eighty six episodes ) leads to muscle atrophy, complex systems are weakened, even killed when deprived of stressors.

-Nassim Taleb, Antifragility (draft of prologue)

Such protectionist policies enforce stability at the cost of stifling both resilience and progress. They eliminate the checking process essential to trial-and-error learning, the way by which we identify the “failures” that new forms might correct.

-Virginia Postrel, The Future and Its Enemies

Google’s server architecture is very robust against failures. The quality of the company’s products, and their bottom line, depend on their ability to process enormous amounts of data without interruption and with a low risk of losing any of it. The danger is not hypothetical–companies have been wiped out because some freak accident they were unprepared for destroyed a large fraction of the data they relied on.

Steven Levy’s book on Google makes it clear that they were forced to become robust by their circumstances. Most companies at the time would pay for expensive, high-end servers that had a very low rate of failure. Google did the opposite–they went for inexpensive servers with an extremely high rate of failure. In order to survive, they had to create software for their servers that would preserve their data and keep their workflow from being interrupted even as servers failed left and right.

Google owes their resilient infrastructure to the fragility of their early servers.

In an active quest for resilient infrastructure, Netflix imposed disorder by design upon their servers.

Imagine getting a flat tire. Even if you have a spare tire in your trunk, do you know if it is inflated? Do you have the tools to change it? And, most importantly, do you remember how to do it right? One way to make sure you can deal with a flat tire on the freeway, in the rain, in the middle of the night is to poke a hole in your tire once a week in your driveway on a Sunday afternoon and go through the drill of replacing it. This is expensive and time-consuming in the real world, but can be (almost) free and automated in the cloud.

This was our philosophy when we built Chaos Monkey, a tool that randomly disables our production instances to make sure we can survive this common type of failure without any customer impact. The name comes from the idea of unleashing a wild monkey with a weapon in your data center (or cloud region) to randomly shoot down instances and chew through cables — all the while we continue serving our customers without interruption. By running Chaos Monkey in the middle of a business day, in a carefully monitored environment with engineers standing by to address any problems, we can still learn the lessons about the weaknesses of our system, and build automatic recovery mechanisms to deal with them. So next time an instance fails at 3 am on a Sunday, we won’t even notice.

Netflix understands that failure is feedback. Until something goes wrong, they won’t be able to figure out what problems exist in their ability to cope with failure. So rather than resting no their laurels, they put themselves through a constant trial by fire to force themselves to be ready and improve their system. It is no different than getting small doses of a disease or poison in order to build an immunity, or working your body out above and beyond the demands your life makes on it in order to increase its fitness. There are many things in human life where stressors are a prerequisite for improvement–or simple maintenance.

Yet stressors are precisely what we seek to hide from in the world of policy. It is my contention that we are too terrified of short term risk and volatility in this country. Rather than embracing Chaos Monkeys of our own, we simply keep a spare in the back of the car and assume everything will go well if we ever have a flat. The only way to grow stronger, wealthier, and more resilient in the long run is to expose ourselves to a lot more risk and volatility than we have lately shown a willingness to cope with.

Deafening Ourselves

It’s not my purpose to single out the environmental movement, but that does embody a certain mentality about risk that has become so tied up in intellectual knots that it has the net long term effect of making things more risky. It is my thesis that a small number of people have to be willing to shoulder greater risks in order to create changes that eventually reduce risk for civilization as a whole.

Solve for X: Neal Stephenson on getting big stuff done

Stephenson’s point about risk is part of his larger argument that innovation in this country has stagnated, a view he shares with Tyler Cowen and Peter Thiel, among others. Putting his general conclusion to the side, I think the importance he places on at least some subset of the population needing to shoulder more short term risk to reduce overall long term risk is absolutely true.

Instead, we take measures to “manage risk”, deafening ourselves to feedback in the process.

For example, there are risks associated with allowing people to build what they want on the property that they own. They could introduce something that disrupts the neighborhood, either by taking up all the parking, or making noise, or both. So we have zoning laws, building permits, and various business licenses. As a result, real estate supply cannot respond to the massive demand for city living, and prices skyrocket.

Moreover, fewer business experiments are possible when everything has to fit a cookie-cutter business license. In Fairfax County, Virginia, a small theater had to wait nearly a year to open because the county had never had a theater before and wasn’t sure how to license one. That’s an enormous opportunity cost to impose on an operation of that size.

The political process through which license or zoning categories can be changed, and permits are issued, is extremely slow to respond to changes on the ground. While a more open system would hear the demand for denser development as loud as a scream, we’re so busy protecting ourselves from short term disruptions that we have essentially left ourselves deaf to it, and to all the potential beneficial innovations that could have happened.

This is no academic point; the toll of this aversion can be measured in wealth as well as lives. Nothing is more emblematic of our attitudes towards risk than the 12 year, multimillion dollar process that new drugs must go through before the FDA allows them to go to market. This lag has led to countless unnecessary deaths (PDF), not to mention making new drugs enormously more expensive once they finally do reach the market. And the ability of the FDA trials to even truly keep us safe is questionable–the data are not really random, and any effect that might seem small for a sample of thousands might never the less effect a huge number of people once it hits a market of millions.

The bottom line is that there are things that cannot really be known until you take the drug to market. Doctors should have to perform their due diligence of informing patients of the risks and unknowns, but delaying entry by over a decade and piling on enormous costs accomplishes very little. Unless your goal is to drastically reduce the number of new treatments we are capable of discovering per year.

We put off the short term risks and increase our long run costs.

Ditch Stability

The economy, politics, and job market of the future will host many unexpected shocks. In this sense, the world of tomorrow will be more like the Silicon Valley of today: constant change and chaos. So does that mean you should try to avoid those shocks by going into low-volatility careers like health care or teaching? Not necessarily. The way to intelligently manage risk is to make yourself resilient to these shocks by pursuing those opportunities with some volatility baked in. Taleb argues— furthering an argument popularized by ecologists who study resilience— that the less volatile the environment, the more destructive a black swan will be when it comes. Nonvolatile environments give only an illusion of stability

-Reid Hoffman and Ben Casnocha, The Start-up of You

We need more risk and volatility, and we need to give up our fruitless quest to hide from them.

In many ways this quest reflects a lack of historical perspective. We bail out the US automakers again and again because they were once the symbol of American greatness, and we think that once they are gone we will never shine again. Yet we forget that at the turn of the 20th century, 41 percent of our labor force was employed in agriculture, and at the end of it, it was down to less than 2 percent. We have undergone massive sectoral shifts before. There is no guarantee that it will go as well this time, but there’s also no reason to think that it won’t.

We restrict immigration and imports because they pose an immediate risk to specific workers and businesses in the short run. Yet we forget that during periods of far more open immigration and trade, we experienced historically unprecedented levels of growth. Moreover, opening these channels opens us to feedback–from the ideas, new business models, the scientific and technological breakthroughs occurring worldwide and that might occur here if we would allow people to come here.

We should not be focusing our efforts on fighting risk and volatility, but on fighting fragility. We should fight for feedback.

It is only in the face of volatility that we are able to innovate and grow resilient.

Cultural Innovation — Putting Together the Pieces

My goal in 2012 is to write at least one paper and try to get it published. The paper I have in mind is inspired by three men, and their corresponding books. These are Friedrich Hayek and The Constitution of Liberty, Thomas Sowell and Knowledge and Decisions, and Everett Rogers and Diffusion of Innovations. I want to put the pieces together in order to make a single, solid argument, but I suspect I’m going to need a few more pieces before I can get there.

F. A. Hayek: Trial and Error and Local Knowledge

At any stage of this process there will always be many things we already know how to produce but which are still too expensive to provide for more than a few. And at an early stage they can be made only through an outlay of resources equal to many times the share of total income that, with an approximately equal distribution, would go to the few who could benefit from them. At first, a new good is commonly “the caprice of the chosen few before it becomes a public need and forms part of the necessities of life. For the luxuries of today are the necessities of tomorrow.” Furthermore, the new things will often become available to the greater part of the people only because for some time they have been the luxuries of the few.

-Friedrich Hayek, The Constitution of Liberty

Hayek argued that everything in human society–from technology to words to ideas to norms–begins its life as something developed and adopted by a small subset of the population. Some tiny fraction of these end up gaining mainstream adoption.

When I read The Constitution of Liberty two years ago, I became enamored by this very simple framework. It seemed an elegant explanation for how cultures evolve over time, through a process of rote trial and error.

On the other hand, I found the fact that Hayek didn’t elaborate on the process any further to be frustrating. If I had my way, I would throw out every last section of that book except the bits on cultural evolution, and have had him make up the other 400 some pages by digging deeper into this concept.

What Hayek is known for more widely is his work on local knowledge. In particular, “The Use of Knowledge in Society” discusses how the price system makes it possible for people to act on their specific knowledge of time and place without needing to get the much more difficult to acquire big-picture knowledge. Speaking of a hypothetical man on the spot, he wrote:

There is hardly anything that happens anywhere in the world that might not have an effect on the decision he ought to make. But he need not know of these events as such, nor of all their effects. It does not matter for him why at the particular moment more screws of one size than of another are wanted, why paper bags are more readily available than canvas bags, or why skilled labor, or particular machine tools, have for the moment become more difficult to obtain. All that is significant for him is how much more or less difficult to procure they have become compared with other things with which he is also concerned, or how much more or less urgently wanted are the alternative things he produces or uses. It is always a question of the relative importance of the particular things with which he is concerned, and the causes which alter their relative importance are of no interest to him beyond the effect on those concrete things of his own environment.

Hayek’s entire worldview was built around the idea of complex human systems which required more knowledge than any one individual within them could possibly have, something that Leonard Read captured more poetically in “I, Pencil“. The process of cultural evolution involved individuals and small groups trying out something new, which is observed by others who decide whether or not that new thing fits in with the particulars of their own circumstances, needs, and taste. In short, it doesn’t require much knowledge to come up with something new, and then an incremental amount of local knowledge is brought to bear as more individuals get exposed to that new thing.

But, as I said, he didn’t develop this system in any real detail.

Thomas Sowell: Knowledge Systems

The unifying theme of Knowledge and Decisions is that the specific mechanics of decision-making processes and institutions determine what kinds of knowledge can be brought to bear and with what effectiveness. In a world where people are preoccupied with arguing about what decision should be made on a sweeping range of issues, this book argues that the most fundamental question is not what decision to make but who is to make it–through what processes and under what incentives and constraints, and with what feedback mechanisms to correct the decision if it proves to be wrong.

-Thomas Sowell, Knowledge and Decisions

Sowell begins Knowledge and Decisions by explicitly recognizing his intellectual debt to Hayek in general and “The Use of Knowledge in Society” in particular. Yet in the book he goes far beyond any level of detail that Hayek provided on the subject, at least that I am aware of.

One of the crucial components of the book is the emphasis on feedback mechanisms.

[F]eedback mechanisms are crucial in a world where no given individual or manageably-sized group is likely to have sufficient knowledge to be consistently right the first time in their decisions. These feedback mechanisms must convey not only information but also incentives to act on that information, whether these incentives are provided by prices, love, fear, moral codes, or other factors which cause people to act in the interest of other people.

Clearly, feedback mechanisms must play a huge role in Hayek’s process of social trial and error. Feedback mechanisms are what determine what is considered “error” and force people to change course. As Sowell explains, they take many forms:

A minimal amount of information–the whimpering of a baby, for example–may be very effective in setting off a parental search for a cause, perhaps involving medical experts before it is over. On the other hand, a lucidly articulated set of complaints may be ignored by a dictator, and even armed uprisings against his policies crushed without any modification of those policies. The social use of knowledge is not primarily an intellectual process, or a baby’s whimpers could not be more effective than a well-articulated political statement.

He added “[f]eedback which can be safely ignored by decision makers is not socially effective knowledge.”

So discerning what outcomes we should expect from the various forms of social trial and error requires identifying the relevant feedback mechanisms. The feedback that potential new words faced takes a very different form than the feedback a new product on the market faces, or a publicly funded project.

The particulars of these feedback mechanisms, along with the incentives and institutional context, determine “what kinds of knowledge can be brought to bear and with what effectiveness” in each given case.

In many ways, Knowledge and Decisions is just good old-fashioned economics–it deals with incentives, with inherent trade-offs, and with scarcity. But it is a particularly Hayekian take on economics, with its focus on the scarcity of knowledge in particular and the role of very localized, difficult to communicate knowledge.

I don’t think Sowell gets nearly enough credit for this work among economists generally or even among Hayekians.

Everett Rogers: Curator of His Field

This book reflects a more critical stance than its original ancestor. During the past forty years or so, diffusion research has grown to be widely recognized, applied, and admired, but it has also been subjected to constructive and destructive criticism. This criticism is due in large part to the stereotyped and limited ways in which many diffusion scholars have defined the scope and method of their field of study. Once diffusion researchers formed an “invisible college” (defined as an informal network of researchers who form around an intellectual paradigm to study a common topic), they began to limit unnecessarily the ways in which they went about studying the diffusion of innovations. Such standardization of approaches constrains the intellectual progress of diffusion research.

Everett Rogers, Diffusion of Innovations, 5th Edition

After I read Constitution of Liberty, I realized that there was probably a literature behind the kind of phenomena that Hayek was talking about. The term “early adopter”, which has become part of the mainstream lexicon, must have come from somewhere. Hayek was unfortunately of little help; he cited old theorists like Gabriel Tarde. While the diffusion literature owed a certain intellectual debt to Tarde, he was writing nearly half a century before the modern field emerged.

I eventually happened upon Diffusion of Innovations, Everett Rogers’ book, the various editions of which basically bookend the entire history of the field in his lifetime. Which is quite helpful, because it began in his lifetime–and the first edition of the book was instrumental in its formation.

Where Hayek and Sowell’s works are within the confines of high theory, Diffusion of Innovations is a thoroughly empirical book, at times painstakingly so. There is not a single concept that Rogers introduces, no matter how simple, which he does not illustrate by summarizing a study or studies which involve an application of that concept.

Rogers helped formalize many of those concepts himself with the first edition of the book, published in 1962, when the literature was pretty sparse and dominated by rural sociologists. Since then, it has expanded across disciplines and in volume of published works. As a result, in the last edition of the book, published only a year before he died, there were many aspects of the diffusion process that had been solidly demonstrated by decades of work.

The books always served as a tool for both introducing the field to those unfamiliar with it, and attempting to steer future work. In the final edition, Rogers highlights not only what the literature has managed to illuminate, but its shortcomings. In short, the book has just about everything you would want if you were attempting to get a sense for what work has been done and what has been neglected.

There are aspects of the diffusion literature which are quite Hayekian. In particular, the emphasis on uncertainty and discovery processes.

One kind of uncertainty is generated by an innovation, defined as an idea, practice, or object that is perceived as new by an individual or another unit of adoption. An innovation presents an individual or an organization with a new alternative or alternatives, as well as new means of solving problems. However, the probability that the new idea is superior to previous practice is not initially known with certainty by individual problem solvers. Thus, individuals are motivated to seek further information about the innovation in order to cope with the uncertainty that it creates.

The various mechanisms which Rogers describes which individuals employ to reduce uncertainty–trying the innovation on a partial basis, or observing how it goes for peers who have adopted the innovation, or measuring the innovation against existing norms, to name a few–can be seen as clear cut cases of economizing on information.

In many ways the diffusion model that Rogers lays out is the detailed system that I wanted Hayek to develop. Rogers discusses so many specific aspects of the process; such as the role of heterogeneity and homogeneity, people who are more cosmopolitan or more localite, the different categories of adopters–including the familiar early adopters–and on and on. Rogers concisely describes and categorizes the various feedback mechanisms against adoption in the system.

On the other hand, the beginning of the process–the actual generation of the innovation–is where the literature is by far the weakest. Rogers cites several who have criticized it for this, and agrees that it is a problem. He points out several attempts that have been made to address this problem, but it’s clear that not nearly as much work has been done nor are the results as solid.

Part of the problem is the historical origins of the field–the diffusion literature began with rural sociology, where innovations were developed in universities who then peddled their wares to American farmers. The single most influential study dealt with the diffusion of hybrid corn, which seemed very clearly to be a quantifiable improvement over its alternatives. As such, many diffusion studies have the perspective of assuming that an innovation should diffuse, that there is some problem with the people who reject rather than adopt.

How did the pro-innovation bias become part of diffusion research? One reason is historical: hybrid corn was very profitable for each of the Iowa farmers in the Ryan and Gross (1943) study. Most other innovations that have been studied do not have this extremely high degree of relative advantage. Many individuals, for their own good, should not adopt many of the innovations that are diffused to them. Perhaps if the field of diffusion research had not begun with a highly profitable agricultural innovation in the 1940s, the pro-innovation bias would have been avoided or at least recognized and dealt with properly.

Moreover, the outline of what he believes is the process by which innovations are generated is a very directed, top-down process. It involves “change agents” that are consciously attempting to solve problems and diffuse some innovations. I’m not arguing against the existence of such agents–they are obviously an extensive part of society, from medical researchers seeking a cure for cancer and pharmaceutical companies attempting to get their drugs mainstream adoption, to Apple coming up with a completely different kind of smartphone and tablet and bringing them to market.

But the change agents, as Rogers and the diffusion literature envision them, are only a part of Hayek’s story of social trial and error. Consider language–new words and phrases emerge all the time and diffuse through a process which I am certain is identical to the one Rogers describes. On the other hand, I highly doubt that there are “change agents” who developed these new words and phrases in a lab somewhere and then promoted them. I think the process is far more organic.

Rogers also discusses the role of norms in terms of how they hinder or help the diffusion of an innovation, but left unsaid I think is that those norms are themselves undoubtedly the product of a previous diffusion. In Hayek and Sowell’s framework, traditions and existing norms emerged in response to trade-offs that needed to be made throughout a culture’s history. As Edmund Burke put it succinctly in Reflections on the Revolution in France:

We are afraid to put men to live and trade each on his own private stock of reason; because we suspect that this stock in each man is small, and that the individuals would do better to avail themselves of the general bank and capital of nations, and of ages.

The trial and error process that Hayek envisioned built up that “general bank and capital of nations, and of ages” as societies developed increasingly effective ways to manage their trade-offs.

Rogers does touch on this point of view from a couple of angles. First, he describes the work of Stephen Lansing in uncovering the astonishing effectiveness of the local knowledge contained in the religious hierarchy of Bali, as he described in his book Priests and Programmers. This was a case where the seemingly beneficial innovations of the Green Revolution proved inferior to what seemed like mere superstitious practice.

The Balinese ecological system is so complex because the Jero Gde must seek an optimum balance of various competing forces. If all subaks were planted at the same time, pests would be reduced; however, water supplies would be inadequate due to peaks in demand. On the other hand, if all subaks staggered their rice-planting schedule in a completely random manner, the water demand would be spread out. The water supply would be utilized efficiently, but the pests would flourish and wipe out the rice crop. So the Jero Gde must seek an optimal balance between pest control and water conservation, depending on the amount of rainfall flowing into the crater lake, the levels of the different pest populations in various subaks, and so forth.

When the Green Revolution innovations were introduced to the region, crop yields dropped, rather than increased. This intrigued Lansing.

In the late 1980s, Lansing, with the help of an ecological biologist, designed a computer simulation to calculate the effect on rice yields in each subak of (1) rainfall, (2) planting schedules, and (3) pest proliferation. He called his simulation model “The Goddess and the Computer.” Then he traveled with a Macintosh computer and the simulation model from his U.S. university campus to the Balinese high priest at the temple on the crater lake. The Jero Gde enthusiastically tried out various scenarios on the computer, concluding that the highest rice yields closely resembled the ecological strategies followed by the Balinese rice farmers for the past eight hundred years.

Clearly, Balinese society had arrived at this optimal solution through some process. But Rogers does not delve too deeply into this.

Rogers also acknowledges that the literature may have focused too exclusively on more centralized processes.

In recent decades, the author gradually became aware of diffusion systems that did not operate at all like centralized diffusion systems. Instead of coming out of formal R&D systems, innovations often bubbled up from the operational levels of a system, with the inventing done by certain lead users. Then the new ideas spread horizontally via peer networks, with a high degree of re-invention occurring as the innovations are modified by users to fit their particular conditions. Such decentralized diffusion systems are usually not managed by technical experts. Instead, decision making in the diffusion system is widely shared, with adopters making many decisions. In many cases, adopters served as their own change agents in diffusing their innovations to others.

Though recognizing that such processes exist, it’s clear that the work that has been done on this is much thinner than the more traditional, change agent based research.

Questions That Remain

As I said, all three of these pieces have some holes in them, and those holes aren’t necessarily filled just by putting all of them together.

The next logical step would probably be to seek out more material like Rogers’, where a lot of work has been done and concrete conclusions can be drawn. Any work on how new words and phrases emerge and proliferate would probably be a good start.

Online communities also have many customs, such as hashtags on Twitter and the hat tip among bloggers. The advantage to customs like this is that they leave behind recorded evidence, unlike, say, an oral tradition. We know, for instance, when hashtags first became popularized among Twitter users–it is documented. A great deal of work is being done by communications scholars on subjects such as these; this could also probably provide some more solid leads.

What I want to argue is that innovations are generated in a Hayekian trial and error process, and some subset of them gain mass adoption in the manner described by the diffusion of innovations literature. I want to describe the role that local knowledge plays in that process; how the feedback mechanisms and incentives shape what innovations are generated and which ones ultimately are adopted.

But there’s more research to be done before I can make a case for this thesis that is solid enough for me to be comfortable with.

The Diffusion of Innovations

One kind of uncertainty is generated by an innovation, defined as an idea, practice, or object that is perceived as new by an individual or another unit of adoption. An innovation presents an individual or an organization with a new alternative or alternatives, as well as new means of solving problems. However, the probability that the new idea is superior to previous practice is not initially known with certainty by individual problem solvers. Thus, individuals are motivated to seek further information about the innovation in order to cope with the uncertainty that it creates.

-Everett Rogers, Diffusion of Innovations, 5th Edition.

No one can pretend to a comprehensive understanding of human social systems until they have read the latest edition of Everett Rogers’ Diffusion of Innovations, or familiarized themselves with the literature it surveys by some other means. This is not to say that you will achieve perfect knowledge of such systems upon completing the book, nor would Rogers have made such a claim. What Rogers provides is a sense of how much has been accomplished in the young field he helped to create, and how many unanswered questions still remain.

The Basics

Image from UNODC

Rogers is relentless in his categorization and definition of concepts in the diffusion model, but some basic notions can be spelled out without resorting to his level of detail.

The contribution of the diffusion literature that has itself diffused widely beyond the field is the concept of the early adopter. Rogers lays out several categories of adopter, including a stage before the earlier adopter, which he calls the “innovator”. Counterintuitively, the innovator is not actually the one who comes up with the innovation, but is simply the very first to adopt it. Then comes the early adopters, followed by the early majority, the late majority, and the laggards.

Studies conducted in different disciplines across a broad range of subjects over a course of decades have consistently found that adoption, plotted over time, looks like the S-shaped curve pictured above. The initial adoption period, during which only the innovators and the early adopters are adopting, begins relatively slowly. The middle period, when the early majority and then the late majority adopt, occurs extremely quickly. Finally, the laggards are the last to the party and even after everyone else has adopted their adoption is quite slow.

The part of the diffusion curve from about 10 percent adoption to 20 percent adoption is the heart of the diffusion process. After that point, it is often impossible to stop the further diffusion of a new idea, even if one wished to do so.

Heterogeneity and homogeneity are crucial components of the social system in which innovations spread. Innovators are standalone individuals who are so different from the rest of the people in the social system that their adoption does nothing to encourage other individuals to adopt the innovation. I used to think that Robert Scoble was the quintessential early adopter, but by Rogers’ terminology I think he is actually an innovator. He tries out absolutely everything, often years before anyone else does. The way he uses the innovations he adopts is often very different from how much later adopters will end up using it. I think few people actually adopt something because the Robert Scobles of the world did it first.

On the other hand, early adopters are different enough that they are more likely to adopt an innovation than the majority, but similar enough to the majority that they are much more likely to follow suit eventually.

Early adopters are a more integrated part of the local social system than are innovators. Whereas innovators are cosmopolites, early adopters are localites.

In spite of what many early 20th century communications thinkers believed, mass media has an insignificant effect on our behavior compared to our peers. Once adoption reaches the “majority”–which Rogers claims accounts for about 68 percent of a population–the adoption rate skyrockets, because the early and late majorities are comprised of a large number of highly homogeneous individuals who are looking to one another for cues about whether the innovation is worth the effort of adopting.

The laggards are the last group to adopt, and their rate of adoption remains quite slow relative to the big upsurge in the middle period of diffusion. One interesting thing I had not realized before reading Diffusion of Innovations is that there usually exists a big socioeconomic gap between early adopters and laggards. This makes a certain sense–the downside risk is smaller for a relatively wealthier person taking on the costs of adopting an untested innovation than it is for a relatively poorer one. The difference between the two categories isn’t always one of wealth, though, but the gap usually does exist in some form of social status.

There is much, much more to it than this basic picture. An enormous amount of work has been done studying the various communication channels through which innovations spread, and the social systems that provide the institutions and context the adopters interpret the innovations from, and just about every aspect of the innovation-generation, diffusion, and implementation processes.

The Book and Its Author

Diffusion of Innovations is an interesting book with an interesting history. I don’t know if it’s really accurate to call what I read the 5th edition, as I get the sense that it is so radically different from the first as to be almost an entirely distinct book. The first edition was published in 1962, with the express intention of unifying the research efforts conducted in disparate academic disciplines and providing a common theoretical framework. The 5th edition was published in 2003 and is just as concerned with criticizing and exposing the flaws in what Rogers calls “the classical diffusion model”–the one he himself was instrumental in formalizing!–as it is with introducing the basics to newcomers.

Rare is the scholar who introduces a theoretical model that becomes the foundation for an entire line of academic research. Rarer still is the scholar who is able to see various critiques and contradictions to his model, accept them, and work to improve the model! Rogers was exceptionally open minded and well read. On the latter count, every theoretical concept introduced in the book is immediately followed up with specific case studies to demonstrate what they mean in practice. He was there at the very beginning of diffusion research and has lived to see it evolve.

My only regret where the book is concerned is that it was not written more recently–at the time it was published, Rogers’ most recent source of information on Internet adoption showed that there were a little over 500 million computers connected to it in the world. Today, over 800 million people are active on Facebook alone! From his Wikipedia page I see that Rogers passed away in 2004–this is truly tragic, as I would have loved to read what he thought about what has transpired on the web in the 9 or so years since the 5th edition was published.

Any scholar of human nature who isn’t familiar with this literature owes it to themselves to read this book.