Innovation will Bubble Up from the Long Tail

I recently wrote that the long tail of digital content producers–that is, the vast majority–will make nearly nothing in revenue. This is especially true when compared to the head of the tail, the tiny fraction of content producers that will earn the vast majority of the revenue. By this I did not mean that the long tail was unimportant–in fact, I believe that the long tail is the most important segment, because that is where the future can be found.

Social Trial and Error

Societies progress through continual and parallel processes of trial and error. Small groups adopt products or activities or norms, a subset of which are picked up by larger groups, and even smaller subset of which goes on to yet a larger group. This process continues until only a tiny fraction of the original products, activities, or norms go mainstream.

An enormous number of trials end up discarded before a single one makes it to even a middle level of adoption, much less the favored few that go mainstream–or stay there long.

This phenomena, well documented in the diffusion of innovations literature, is most familiar to people in the fast moving world of consumer technology, where phrases like “early adopters” are used in casual conversation. It applies to anything that could proliferate across groups–for example, art, and content more generally.

A Hotbed of New Ideas and Failure

An unknown, aspiring writer in today’s world faces the same problem as any unknown, aspiring writer did in generations past–obscurity. He has many more tools at his disposal than his equivalents in the past did–he can start a blog, podcast, and connect with others on social networks to promote his work. There are also many, many more places he can submit his work–there are still magazines in the traditional sense, but there are many more online outlets of widely varying audiences.

These tools are available to any aspiring writer, however–in fact, the barriers to putting out your writing in public are so low that huge swaths of people who wouldn’t have even tried in the past are also putting their stuff out there. If anything, the web and the new opportunities it affords have actually reduced the probability that any one aspiring author will make it big.

If they want to set themselves apart, they will have to innovate. Of course, as touched on above, most of these innovations will fail to gain any traction. The new and exciting things happening in writing, however, will come from the successful subset of these innovations.

Scott Sigler is an example of a successful innovator in digital writing. After losing his book deal ten years ago, he learned about podcasting and decided to record and serialize his book himself, and put it out for free. He continued to do this after the first book, and eventually had built up a big enough audience to catch the attention of Dragon Moon Press, a small independent publisher. On the strength of his online following–who helped not only with sales but with marketing the book–the book managed to rocket up Amazon’s bestseller list. This caught the attention of Random House, with whom he currently has a contract. His second book with them was a New York Times bestseller. He isn’t selling Harry Potter-level blockbusters, but he has definitely moved up out of the long tail and into the head.

Sigler wasn’t the sole creator of the podcast novel; others were trying it out at basically the same time. But Kevin Kelly has documented how innovations and ideas tend to occur in parallel in art, as well as in science, math, and technology.

The podcast novel is a great example of how this dynamic works, too, because while it helped launch the careers of Sigler and some of his peers, the form itself has yet to become anything like mainstream. It has grown over time, but, one of the largest repositories for podcast novels on the web, still boasts a mere 569 titles, and a registered audience of 83,000. If you asked a random individual, even a random book enthusiast, odds are extremely low that they would have even heard of podcast novels.

Most innovations never make it even this far–but there is still no guarantee that podcast novels will get to the level of mainstream adoption, or even mainstream awareness.

The Head of the Tail is Conservative

The big record labels, publishing houses, and movie studios will never try something truly new. I am confident on this point–anything that is celebrated as being new from these big institutions will in fact just be the first time that big money has been spent on a form that was tried out first by individuals in the long tail.

It makes sense–a lone writer, musician, or filmmaker is working with a very small budget. In a writer’s case, they may pay next to nothing and face an opportunity cost made up primarily of their time. If they try something new and different and it fails, they may be out a few months of work. A big publishing house, on the other hand, has to pay the salaries of its army of editors, not to mention the costs of promoting a work. In dollar terms at least, failure hurts a publishing house a lot more than it hurts the lone, unknown author.

And publishing houses still fail more often than they succeed. They just win really big when they do win, and that subsidizes the failures. Profitably depends on their ability to increase the fraction of the authors they sign on who end up being successes, and minimizing the failures.

For that reason, they are always going to stick to the tried and true. Innovations will have to gain widespread adoption in the long tail–and for a while–before they bubble up to the head.

Consider the movement towards ebooks. The formats that Amazon, Barnes and Noble, and Apple are providing consumers are essentially nothing more than the digitization of the print versions. They do not offer the increased capabilities that digital technology makes possible–of including video and audio files mixed in with the text, for example, something commonly done on blogs. They optimize for the tried and true, because all the money is being invested in the tried and true.

Mechanisms exist for making money from innovations–you could pitch an idea beforehand on Kickstarter, or make an app for smartphones and tablets and charge a price for it. Only after a respectable amount of money has been made by innovators will the institutions at the head of the tail start to take notice.

So while I don’t agree with Chris Anderson’s original hypothesis that the long tail will be of increasing monetary significance to businesses, I do think that it will be an even greater engine of innovation in the digital era than it was in the analog one.

Standing at the Tip of the Iceberg

Who Says What to Whom on Twitter is an absolutely fantastic piece of scholarship. It received a fair amount of visibility among the tech press for one of its findings–that 0.05 percent of Twitter users account for 50 percent of “tweets consumed”–IE, tweets that appear in people’s timelines. This was actually the least interesting finding of the study as far as I was concerned. Anyone who had read Clay Shirky’s essay on power laws would have anticipated that kind of disproportion.

What interested me about the study was first and foremost the methodology–the authors thought through very carefully how to go about gathering data, classifying the different users, and what the flaws of their approaches were. They then took measures to adjust for those flaws. For example, they used two separate methods for categorizing the most followed users in their sample. Each method had completely distinct biases, so the fact that they both yielded very similar results made a strong case that the authors were on the right track.

The study put the “elite” (highly followed) Twitter accounts into four categories–celebrities, media, organizations, and bloggers. Interestingly, they found that each category mostly followed and retweeted members of their own category. I immediately began to think of the economics blogosphere, which involves a handful of very widely read bloggers who all read and engage with one another. As Paul Krugman recently put it:

Twenty years ago it was possible and even normal to get research into circulation and have everyone talking about it without having gone through the refereeing process – but you had to be part of a certain circle, and basically had to have graduated from a prestigious department, to be part of that game. Now you can break in from anywhere; although there’s still at any given time a sort of magic circle that’s hard to get into, it’s less formal and less defined by where you sit or where you went to school.

Emphasis added by me.

Most bloggers these days also have Twitter accounts–this pattern also holds for the economics blogosphere. I would love to use the methodology adopted by the authors of “Who Says What to Whom on Twitter” in order to find and categorize the members of Krugman’s “magic circle” in order to more closely study the dynamics of that circle. The economics blogosphere is a microcommunity, a subset of the larger web and Twitter community–and within that microcommunity are people who more closely follow specific members of the circle than others, who are into Austrian economics, or monetary economics, or development economics. Observing how the online economics microcommunity coheres and fragments, in comparison to the larger view that the paper takes, would be very interesting indeed.

Part of what is so exciting about reading a paper like “Who Says What to Whom on Twitter” is both the sense of a larger media literature that spans decades, and the sheer newness of the field of social media research in particular. There is still so much unexplored territory. In attempting to answer the “what” of the titular question, the authors are forced to limit their focus to the New York Times‘ Twitter account, since the content that they tweet has been categorized already by what section of the newspaper it is in. This is a novel approach, but one the authors admit is quite limited.

How could one possibly categorize content on a larger scale? The authors mention another study that uses Amazon’s Mechanical Turk to this end, but argue that this approach does not scale well. I think that the companies in a position to do this right are the big search engines. Google and Microsoft get millions and billions and queries a day, and they record the specifics of what keywords people put in and what links they subsequently clicked. Moreover, they have methods for determining whether the websites they clicked were actually looking for.

Someone working in the Google or Microsoft research departments could take the data from Twitter and categorize the content being linked to by looking at what terms people used when clicking to that content from their search engines (and being satisfied with the result).

The breadth of the knowledge represented by the existing literature on media generally, and social media and the web in particular, is far larger than any one individual could possibly cover comprehensively. And yet, we’ve only just begun to scratch the surface of what can be known in these areas.

I urge those of you reading the sea of blogs on social media out there to take the time, every so often, to read a paper like Who Says What to Whom on Twitter. As Tyler Cowen once said of the economics blogosphere, “We’re just a small number of apes sitting at computers, relative to the overall literature.” There are people out there, like Duncan Watts and his co-authors, who are thinking very hard about the challenges of trying to understand social media and investing time and resources into tackling them.

The Road Ahead for Digital Content

I used to be one of the true believers of the “bits want to be free” school of thought. The fact that the marginal cost of creating another copy of a digital file, along with the cost of distributing that copy, were vanishingly small, meant that in the long run competition would make charging for content an unsustainable strategy.

The success of iTunes had long made me hesitate, but it is the current rise of companies like Netflix and Spotify that have convinced me that paying for content is not an activity that will be ending any time soon.

The Three Paths

Where we are heading, and where we already are to a certain extent, is to a world where there will be three methods of acquiring different content. First, there will be big bundles like Netflix and Spotify, where you pay a flat rate for all you can eat access to gigantic libraries of video, audio, and text. Everything within those bundles, as well as some additional content not available in those bundles, will be available for straight a la carte sales as well, as iTunes has provided for going on ten years now. Finally, the vast majority of content–that is, an amount uncountable orders of magnitude larger than what is available in the bundles or a la carte stores–will be available for free.


Eli has an excellent post on the economics of bundling. In his example, there are a total of 10 songs in a hypothetical market. When sold individually, the rights-holder’s most profitable strategy is to sell them for 50 cents per song. However, when they are sold as one bundle, the per-song cost goes down to 40 cents. I defer the explanation of the logic behind this to his post and the excellent comments on it; if you want to know more about the direction that content industries are headed I highly recommend you take the time to read it in full.

Eli’s example only deals with 10 songs–imagine how low the per-unit cost has been driven for a service like Netflix, which charges 8 bucks a month to access tens of thousands of titles. The fact that users are paying for access, rather than downloading the whole bundle themselves, also allows Netflix to lower the price further.

The digital content industries have only scratched the surface of bundling tactics. We’re getting companies like Netflix and Amazon offering bundled streaming video, and Spotify and Rdio are doing it for music. But I believe it is only a matter of time before someone starts doing it for text. Rumor has it that Amazon is in talks with publishers to create a bundled option for ebooks. This does not surprise me in the least.

Eli suggested that news organizations should stop trying to set up paywalls and instead work on creating a Netflix-like bundle together. We then learned that the Slovakian media was already tinkering with something like this.

The future of bundling in digital content is going to involve both bundles that are much bigger than the current ones, and smaller ones.

The bigger ones will come not just because current bundles will continue to expand their offerings, but because bundle providers will start to mix the categories of content offered in a single bundle. Why should we pay for separate streaming video and streaming music bundles? Why not pay for one bundle that includes both? Or one bundle that includes streaming video, music, access to books and articles, comics, and smartphone and tablet apps? Apple and Amazon are two companies in a position to do just that.

The smaller bundles I have in mind occupy the space between a la carte and the Netflix-like bundles. One kind of smaller bundle that exists right now are albums–an artifact of an earlier era. Apple and Amazon will now let you pay for a whole album, and sometimes a whole season of a TV show, at a lower price than it would cost to buy all of the individual songs or episodes.

But why can’t we pay one price for a band’s entire discography? Or for all of the currently published Dresden Files books, or Walking Dead comics? Or for all of the seasons of a TV series at once?

A la carte

Apple was a true pioneer of a la carte digital offerings with their iTunes music store, which they later extended to include video as well as apps. Amazon has followed in their footsteps, and their Kindle store now dominates a la carte ebook sales.

What Apple did was to drastically reduce the transaction costs of acquiring and listening to music. Before iTunes, the big way to get music to your computer and MP3 player was either to rip it from CDs or to pirate it. Both approaches had transaction costs–you had to have a program to get your music off of your CD, and then have software that took the ripped song files and integrated them into your listening device.

Pirating, while it didn’t cost money, has disadvantages of its own–all the problems one would associate with any black market. First off, you can never be sure of the quality of the file you’re going to get. Secondly, the availability of songs depended entirely on what other people had to offer. There was also always the risk of getting a virus.

The iTunes store had a stable and growing library of titles, it tied the quality of the files to the reputation of a big brand, and it seamlessly integrated with Apple’s own listening device, the iPod. I could go on, but history has shown that consumers saw the advantages of what Apple had to offer and were willing to pay for them.

I think that a la carte libraries will always remain larger than bundles, if just for the simple reason that they will include everything in those bundles. After all, if you are not a Netflix subscriber, the fact that you want to watch one specific movie within Netflix’s library is probably not enough to sway you into signing up. So you can pay for the movie by itself, buying or renting it from Amazon or iTunes.

Then there will be things that content creators do not want to include in the bundles, at least not immediately. When a new season of Doctor Who comes out, the BBC may want to let Netflix subscribers see all the previous seasons to encourage them to watch the current one, but then leave the current episodes in the a la carte stores for the time being.


The vast majority of content will remain available to consumers for a sticker price of zero. The difference in magnitude between what is available a la carte and what is available as bundled content will not be very large. By comparison, the difference in magnitude between what is available a la carte and what is available for free will be more on the scale of difference between a planet and a galaxy, or a universe.

Content industries have always been blockbuster industries, following a power law distribution. Digital technology, the Internet, and the web will only serve to skew that distribution even further–1 percent of the content will account for 99 percent of the revenue, if that isn’t already the case.

Content industries have always had a tiny fraction who were able to make enormous wealth off of their creations, a much larger fraction that could make a decent living, a much, much larger fraction that just used their creations to supplement a primary source of income, and the vast majority who were unable to make a penny off of their creations.

Because of the web and transaction-cost reducing services like app stores, content creators who are unable to make any money off of their creations are for the first time in history able to nevertheless make it publicly available. So that gigantic body of work that will be available for free will come from this category of content creator–and it will comprise what Chris Anderson has called the long tail. Contra Anderson, however, this segment of the market will bring in next to nothing in revenue.

People have and will continue to provide their content for free for a variety of reasons. The most obvious one is that they aspire to make it big–they’re trying to get out of the long tail and into the head of the tail. They can only do so by gaining visibility and building an audience.

Many people who do not desire to make a living off of their creations also make contributions, however. Personally, I do not intend to try and make a living by writing about technology and society. But I write about it both because it is a subject that interests me, and because writing in public helps me connect to others with similar interests. It also provides me with a portfolio I can point potential employers to in order to help them evaluate me. For a more thorough overview of the benefits of making things in public, see Jeff Jarvis’ new book Public Parts.

The availability of free content will continue to put price pressure on a la carte and bundled content at the margin, pushing paid content into even bigger and cheaper bundles. Moreover, more and more new tools will be developed to drive down the biggest cost of free content–its transaction costs. iTunes has driven down the transaction costs not only of a la carte content but of free podcasts. Link sharing on services like Twitter and Facebook drive down the cost of content discovery.

The Content Creator Draft

From the long tail of free content there will emerge a tiny minority of successful individuals who begin to make a living off of what they do. In the world of webcomics, a group of artists are making a living by giving their comics away online for free and making money indirectly, though merchandise, donations, and other similar strategies. Radio host Leo Laporte has managed to successfully make a living by building up a podcast network financed through ads and donations.

From this group, a few will be vaulted up to blockbuster status. Witness Scott Sigler, who began building up a following by podcasting his novels, and then got a book deal with Random House. Or Amanda Hocking, who became so successful selling her books independently in the Kindle Store that she was picked up by a publishing house.

There will remain big intermediaries–like the publishing houses, record labels, and movie studios of today–who will finance the creation of high end content. They used to have an expensive discovery process for finding talent, which for publishers included things like paying “mid-listers” who were not bestsellers but not total flops either. Now, the discovery process will be left entirely to the open web–people will succeed or fail at demonstrating their ability to attract an audience on their own, and the big intermediaries will pick them up only after they have. This reduces the risk to the intermediaries, which in turn reduces the cost of producing high-end content.

So free, a la carte, and bundled content will all be part of the new interconnected digital ecosystem that is growing and maturing around us at this very moment.

The Journey

In 2004, a group of individuals managed to demonstrate that a set of documents that CBS news anchor Dan Rather claimed had been authenticated were in fact forged. These individuals–the bloggers–were not media professionals; they were normal citizens using the new tools for publishing on the web in order to get the word out. They were dismissed by CBS executive Jonathan Klein as each being just “a guy sitting in his living room in his pajamas writing.”

It wasn’t the stakes of the specific documents that was at issue–some frivolous controversy about whether or not then-president Bush had received preferential treatment when he was in the National Guard. What was really interesting about the event was that the once silent audience of mass media was talking back. In the past, if the typical private citizen noticed an error in a public story, they had no means for pointing it out beyond their circle of acquaintances. The web and the applications built on top of it–like blogs–made it possible for people to join the public conversation without going through the filter of media professionals for the first time.

It was that aspect that excited the imagination of one 19-year-old GMU history student, and inspired him to start a blog of his own. One of the big blogs he was paying attention to at the time was Instapundit, so he decided that his blog would have the -pundit at the end, even though he was only vaguely aware of what a pundit actually was. He had been reading about the philosopher Protagoras, one of the Greek sophists, so he decided that the blog would be named Sophistpundit, and realized almost immediately how pretentious that sounded. That didn’t stop him from keeping it.

Four years later, when thinking about how it would be nice to have a place to point potential employers that didn’t include the profanity and melodrama of a 19-year-old, he started a second blog, Cloud Culture, to talk specifically about matters of media and technology. It also had the advantage of having a far less pretentious title.

It has now been about seven years since I started Sophistpundit, and three years since I started Cloud Culture. A lot has changed in that time in my personal and professional life, to say nothing of the web and media. It makes less and less sense to have blogs branded with their own names, and even less sense to continue keeping them separate. I blog in part because I love to think through writing, and to engage in public conversation, but also because I am trying to build up my name. I have no interest in Cloud Culture or Sophistpundit becoming a household name or moderately well known. If I’m promoting something, I’d like to be promoting me, my name.

When I started Sophistpundit, the blog was still the most prominent unit of social media. It was not just a place to write posts, but also to share interesting links, pictures, and videos. These days, we have Facebook and Twitter and their competitors to give us common places for social sharing. Increasingly, my blogs transitioned from places I dropped multiple small updates on a daily basis, to places where I posted long form writing every week, or every other week.

This site will serve as the new home for that kind of content, as well as my personally tailored public profile.