Diffusion is the Engine
Everything that is now mainstream was once obscure. We are now largely aware of this fact, especially those of us who follow technology trends. The “early adopter” is a familiar friend, rather than merely the subject of obscure communications papers.
The study of how the obscure becomes widely adopted began in the mid-20th century in rural sociology. The origin of this field is usually credited to the research of Neal Gross and Bryce Ryan on American farmers’ adoption of hybrid corn in the 1940s. Gross and Ryan relied mostly on structured surveys to gather information on why farmers had resisted the innovation, or chosen to adopt with the particular timing that they had.
A great deal of research continued to be done along these lines, until in 1962 Everett Rogers decided to summarize the findings in the first edition of Diffusion of Innovations. The book was more than a simple account of the literature; it provided a basic theoretical framework, as well as suggestions for the direction of future research. An avid participant in the field throughout his life, Rogers continued to put out new editions of the book to bring it up to date and gently guide future research based on how it had progressed between editions. The fifth and final edition came out in 2003 and included some preliminary research on diffusion patters on the Internet. Rogers died not long after, so there will be no sixth editions summarizing studies on diffusion patterns on Twitter, Facebook, or app stores.
Diffusion of Innovations provides nothing so elegant as a theoretical framework; instead, there is a set of stylized observations, summarized from a large body of research which has developed over more than sixty years. This is not to say that it is without theory—just not the kind of theory you might find in The Wealth of Nations or Civilization and its Discontents. Nothing grand—just a basic framework with a number of theoretical concepts baked in, taken mostly from preexisting works.
One useful concept Rogers and his colleagues honed down is what exactly an “innovation”, the unit of diffusion, is: “an idea, practice, or object that is perceived as new by an individual or other unit of adoption”. With this definition, we don’t need to quibble about what is “really” an innovation or “really” new; so long as it is “perceived as new” by whatever “unit of adoption” we are considering. This is the only manner that the term “innovation” will be used for the rest of this piece.
The diffusion of innovations literature suffered from a short-sightedness due to its focus on specific innovations that were believed by the researchers to be objective improvements over pre-existing alternatives, or in any case an alternative world in which they do not diffuse. Rogers was aware of this, and took his colleagues and his own work to task for what he called a “pro-innovation bias”. Even still, they managed to develop a very informative framework from which to view diffusion.
The hallmark of this literature is the standard s-shaped diffusion curve. When an innovation appears among an observed population, at first it diffuses very slowly among a very small subset of the population. Rogers doesn’t quite spell it out, but this is where a vast supermajority of innovations die. An enormous amount of energy and money has been spent trying to figure out what it is that allows some tiny fraction of innovations to survive beyond this crucial stage, but our knowledge is still very partial. At best, we can tell what increases or decreases the relative probabilities of survival, though what the actual probabilities are we cannot say.
Nevertheless, some innovations do survive. Once they have moved beyond the people categorized as “the innovators” and “the early adopters”, we hit “the early majority”, and the diffusion process takes off like a rocket. As Rogers put it:
The part of the diffusion curve from about 10 percent adoption to 20 percent adoption is the heart of the diffusion process. After that point, it is often impossible to stop the further diffusion of a new idea, even if one wished to do so.
One of the things that increases the probability that one person will adopt an innovation is if it has already been adopted by other people who are a lot like them that they know. The underlying assumption, supported by a variety of research, is that something like 60 percent of every population is homogenous, and innovations spread rapidly once they reach this group. There is a prior group of about 10-20 percent of the population that are quicker to adopt and are usually wealthier or higher status on some margin than the majority. Finally, the remaining population, called (pejoratively—there’s that pro-innovation bias again) “laggards”, only adopt after the majority, and only do so slowly. They tend to be poorer, or older, or lower status on some margin from the majority. The relative percentages of the population made up by each category may vary but there is a stable observed phenomena where there’s the first slow group, the second, biggest fast group which constitutes a majority, and the last slow group.
I believe that this model can be extended far beyond where the researchers have taken it. But to do so, several important questions will have to be answered. While there exists an enormous body of studies on successful and failed diffusions, there are not many studies on what happens after a successful diffusion—the average lifespan of successfully diffused innovations, so to speak. There is some research on how prior diffused innovations lay the foundation for future ones, but I would prefer to see a lot more on this.
I think that the diffusion model describes the dynamics of how norms, traditions, and every human social system emerges. It begins with a set of discrete innovations that build on top of each other until you get path dependence, feedback loops, and ultimately, the large, complex arrangements of the modern world.
But more on the big picture a little further down.
First, let’s look at the small picture.
Networks are the Fabric
Paul Adams’ Grouped provides an excellent summary of the literature on the structure of human networks. And here we mean not “networks” in the technological sense in which the word has come to be used, but networks in the older sense of connections among people. Networks in this sense have existed for as long as human beings have been social animals; which is to say, for as long as they have been human beings.
In the “strong ties, weak ties” literature, we are described as having up to 15 strong ties, and as many as 500 weak ties that we can actually remember something about. Of those, Robin Dunbar has famously found that we are only capable of maintaining stable relationships with 150 people total, including our strong ties. Our weak ties are made up of people who come in and out of that 150 over time, and some never get beyond that threshold.
Of our strong ties, 80 percent of our communications happen with our closest set of up to five connections. Our strong ties exert enormous influence on us, and these five connections are by far the most influential in our lives. A lot of what we like to think of as discrete, individual tastes are really things that people only like because the people they know like them. A great deal—perhaps the majority—of preferences owe their origin to this groupish tendency rather than to some innate internal ranking what someone wants.
All of our friends, even our very closest ones, have their own circles of connections that overlap to greater or lesser extent with ours. The further out into their weaker ties you go, the more unique connections they have that you do not have. The extended connections of your closest connections alone make up many, many thousands of people. These semi-overlapping circles comprise a network that encompasses basically all of mankind, with a few outliers. This is what is meant when it says that everyone can be connected to anyone else within six degrees of separation; and that number has decreased with the Internet and social networks like Facebook.
However, even three degrees is actually a vast social distance. Our personal influence on the group three degrees removed from us is very minimal, compared to the influence we have on our closest five friends, and the influence that they have on us. However, information and influence does come from outside of this inner circle. After all, the people in your inner circle have people in their inner circle that aren’t in yours, who have people in their inner circle that aren’t in your friends’, and so on. Through this network of close connections, information, preferences, norms, slang, and just about every sort of innovation diffuse.
These small networks are the fabric that every social system is stitched from. If one of your five closest connections adopts a certain norm, the odds a very high that you will too—higher, in any case, than if it was just one of your 15 closest connections, or one of your 50 closest, and so on. If one of your weak ties adopts an innovation (especially the 135 or so that you’re able to keep a persistent connection with), it does increase the odds that you will adopt it as well—just not by very much.
People we think of as highly influential have an enormous number of weak ties, but no more strong ties than anyone else. If one of these people adopts an innovation, there’s a small probability that their weak ties will. But that small probability still constitutes an increase from before they had adopted it—and every so often something will take off among their weak ties.
Nevertheless, Paul Adams is firm on the point that the overwhelming majority of influence occurs within the loci of these small networks; regular people influencing other regular people who constitute their closest connections; and those people influencing yet other people who constitute some of their closest connections. We influence, and are influenced in subtle, almost invisible ways, by our closest connections. Agenda setting, and other top-down models of influence, are bogus.
Diffusions Accumulate into Traditions
We can think of these small networks as going through an endless trial and error process, in which the vast supermajority of innovations are rejected as errors. Of those innovations that do diffuse more widely, the overwhelming majority do not survive longer than a year (we call these “fashions”). This is clear from the logic of the Lindy Effect; first formulated by Benoit Mandelbrot and lately popularized by Nassim Taleb. The Lindy Effect states that every technology (understood in the broadest possible sense) is on average halfway into its lifespan; meaning that new innovations are almost always a flash in the pan.
The Lindy Effect also makes it clear that any innovation that has lasted a long time, will last a long time yet. So some tiny fraction of initial innovations diffuse, and a small fraction of those innovations survive their first year post-diffusion. Over time, these survivors accumulate to form the basis of our norms and traditions.
Michael Oakeshott described specific traditions as voices in the conversation of mankind, but I think this provides too great a sense of uniformity within each tradition. To me, each tradition is more like a conversation in itself, that has been going on for a very long time, and whose participants fluctuate within a given time span and across generations.
These conversations are the product of an inconceivably large number of previous diffusions. Over time, traditions develop their own vocabularies, styles of discourse, and patterns of behavior. These emerge as the result of specific innovations—specific terms, specific discourses that exhibit particular characteristics, specifics behaviors—diffusing, each diffusion increasing the probability that future successful innovations will be related to those that have already been adopted.
The conversation metaphor should not imply that traditions are primarily composed of words and articulated concepts. Though each tradition does have a set of articulated stories, these are a small fraction of the whole of a tradition, which is primarily made up of unarticulated, tacit, practical knowledge. We know when we have made a faux paus within the context of a particular tradition even if no one explicitly tells us so, because our fellow participants have a wide variety of ways of signaling this to us. More to the point, we are wired to keep an eye out for signs that we have breached the etiquette of the circumstances, especially when we are very young.
Where one conversation begins and another ends is not quite clear; there is the Christian tradition but then there is also the Protestant tradition, and the Calvinist tradition, and the American Presbyterian tradition. In some ways these traditions represent subsets of a larger whole, but in other ways they are divergent branches from a common origin.
Each conversation is maintained through an apprenticeship-like passing on of the practical knowledge required to continue it to members of the latest generation. Each new initiate takes from the store of practical knowledge and stories that grew from the conversation that had occurred before their birth, and each puts something back into it—though most of these contributions end up being forgotten by time entirely. Moreover, what each individual believes to be worthy of putting back into the conversation they may think they arrived at of their own volition, but in reality they were guided there by the way in which their perceptions were framed by the tradition itself. The most important part of making any decision is what information we choose to filter out or focus on as significant, and this is precisely the role played by the context that any tradition provides us. This is the reason why parallel innovations frequently occur not only in the math and sciences, but in engineering and even in art. Human talent is funneled down the relatively narrow corridors provided by the traditions they are embedded within.
Stories, practices, norms, fashions, lifestyles, and all forms of human behavior diffuse within and across these conversations. Each innovation has a non-zero probability of spreading within a given conversation. Different factors increase or decrease this probability. Within current mainstream economics, a new paper is more likely to draw attention if it makes what economists consider to be a novel point, and does it within the context of the neo-classical, Samuelsonian, rational expectations model of the world. It may be a criticism of said model, but coherence to those who practice scholarship in the context of being trained in that model will increase the probability of diffusion.
In short, coherence within the framework of a particular conversation increases the probability of diffusion within that conversation, while relative difficulty to comprehend an innovation within the context provided by a conversation decreases its odds of diffusion within that conversation. For this reason, the very things that make low-probability innovations unlikely to diffuse make them the most disruptive when they do; for it is precisely because they are so far from comfortable points of reference within a tradition that they are unlikely to be adopted.
Oakeshott argued that the various conversations of human social life are entirely independent and have distinct characteristics. It is difficult to think of a version of reality where this is actually the case. The diffusion of innovations literature demonstrates that the more like one other people are, the more likely diffusions are to spread among them. The fact that I have been exposed to the conversation of economics, among others, does not change the fact that I grew up in northern Virginia, in America, in an English-speaking middle class family, and went through the Virginia public school system from kindergarten through grad school. There are a lot of people who are similar to me on some or all of these dimensions but don’t know anything about economics or many of the other conversations that I participate in. Yet there is an enormous body of traditions that we share in common.
As a part of our small networks, we all form links across the respective conversations that we participate in. For this reason, while innovations are more likely to diffuse within a conversation than across them, the larger super-conversations are large enough to bridge the conversations quite frequently. So while it has become highly apparent to me that the Paleo Diet has spread like wildfire among the libertarian community, it did not originate there, and it is a popular diet beyond the confines of this particular subset of the population.
Nassim Taleb has expressed concern that what autonomy these conversations do have is being jeopardized by globalization and by the Internet. Drawing on the logic of island biogeography, he fears that reducing this autonomy will reduce the overall diversity of conversation ecosystem. This is no trivial concern, as maintaining the diversity of these traditions requires maintaining a diverse set of practical knowledge that it is not at all clear we can ever get back if they are lost. Taleb approaches this from a risk-management perspective: we do not know what practical knowledge we will need in the future for events and changes in circumstance that we are unable to predict today. As such, more diversity is better, as it hedges our bets more effectively.
It certainly seems clear that increased interconnection has led to more highly skewed power law distributions. However, this seems to be offset, in the sense Taleb is concerned with, by an ever-lengthening long tail of conversations. So while there are more rock stars that everyone is aware of, more and more niche conversations are taking place, developing their own terminology, practices, stories, and frames of reference. In academic life, while cross-discipline work is trendier than ever, the fact remains that the long term trend has been towards ever greater specialization, branching out from older, larger traditions and forming their own niche conversations. Taleb’s concerns need to be taken seriously, but there is some reason to hope.
Within the Network and Across Traditions
We all live within the small network point of view; we see the world through our direct experience, which includes the people that are a regular part of our lives. Up to five such people dominate the majority of our social interactions, and the Internet has only empowered this further since we can now communicate with people via chat, social networks, and email during work hours.
From this perspective, the stream of innovations seems like a constant barrage. We can mostly tune out the stuff diffusing through our weak ties but the ones that make it to our strong ties, and especially our five closest connections, are very difficult for us to resist. Most of these are a flash in the pan and are gone after a brief visit to our lives. Many of them remain relatively obscure, and some of them go viral to such an extent that there seems to be no one in our lives who has not at least heard of them.
Another part of the human experience are the various Oakeshottian conversations we participate in—that is, the traditions within which we are embedded, and the communities we are a member of that operate within those traditions. Though traditions are frequently seen as static, stagnant burdens on individuals living within the modern tide of change, they are not only dynamic, but highly volatile. The volatility comes from the large number of innovations that diffuse within them every year.
Of the small fraction of innovations that are introduced to a tradition which manage to diffuse, most are quickly weeded out by feedback mechanisms built into the tradition. This feedback is the cumulative result a huge number of prior diffusions which built upon one another to develop criteria for what sorts of innovations would be considered for adoption, and what sorts of consequences would result in rejecting innovations that had previously been adopted.
More important than feedback is time, the impact of which is similar to but broader than any one feedback mechanism. Over time, circumstances change; most innovations do not survive beyond the initial circumstances that were favorable to their adoption. Over time, current participants are swapped out for new ones, and the transmission of a tradition to the next generation may not include all of the innovations that had been adopted into it by the current or previous generations.
Our networks guide our actions in subtle, nearly invisible ways, and these networks are embedded within a set of traditions. The traditions are always fluctuating with innovations diffusing within and across them, and our networks form the connective tissue across which these innovations diffuse.
This constant process of diffusion, rejection, and persistent adoption constitutes an ongoing process of trial and error through which all human social systems evolve. I am tempted to say “progress” but it is difficult to define positive advancement for a system that involves changing norms and moral rules. Even from a material standpoint there will be more dead ends than successes, and many dead ends won’t even become apparent until generations after the fact. For all we know, the changes that birthed the great discontinuity in standards of living at the onset of the Industrial Revolution will turn out to be fatal to us in the long run.
We are not guides, arbiters, or engineers of this process. Our individual contributions are small and fleeting in the face of the enormous and persistent edifice of the traditions we are a part of, and the scale of the population living and dead that participate and have participated in them. We are, in short, mostly along for the ride.
