Rereading The Long Tail

This was officially launch week for The Umlaut, a new online magazine that my friends Jerry Brito and Eli Dourado have started. There are five of us who will be regular writers for it. For my first piece, I thought it might be fun to go back and re-examine The Long Tail almost seven years after it was published.

The Long Tail had a big impact on the conversation around new media at the time, and was very personally significant. The original article was published in October of 2004, a mere month before I began blogging. Trends in new media were a fascination for me from the beginning, so I kept up with Chris Anderson’s now-defunct Long Tail blog religiously.

A 19-year-old and a tad overenthusiastic, I strongly believed that the mainstream media was going the way of the dinosaur and would be replaced by some distributed ecosystem of mostly amateur bloggers. In short, I thought the long tail was going to overthrow the head of the tail, and that would be that. Moreover, I thought that all content would eventually be offered entirely free of charge.

That was a long time ago now, and my views have evolved in some respects, and completely changed in others. I think that the head of the tail is going to become larger, not smaller, and professionals are here to stay–as I elaborate on here. However, I do think that the growth of the long tail will be very culturally significant.

When I began rereading The Long Tail, I expected to find a clear argument from Anderson that he thought the head of the tail would get smaller relative to the long tail. Instead, he was frustratingly vague on this point. Consider the following quote:

What’s truly amazing about the Long Tail is the sheer size of it. Again, if you combine enough of the non-hits, you’ve actually established a market that rivals the hits. Take books: The average Barnes & Noble superstore carries around 100,000 titles. Yet more than a quarter of Amazon’s book sales come from outside its top 100,000 titles. Consider the implication: If the Amazon statistics are any guide, the market for books that are not even sold in the average bookstore is already a third the size of the existing market—and what’s more, it’s growing quickly. If these growth trends continue, the potential book market may actually be half again as big as it appears to be, if only we can get over the economics of scarcity.

Let us unpack this quote a little.

First, Anderson is offering the fact that more than 25% of Amazon’s book sales occur outside of its top 100,000 titles as evidence of the revenue potential for the long tail. But this is very flawed conceptually. At the time of the book’s publication, Amazon sold some 5 million books. If nearly all of the additional revenue beyond the top 100,000 titles was encompassed by the following 100,000 titles, then 4% of Amazon’s titles account for nearly all of its book revenues. And there is good reason to believe that that is exactly how the distribution played out, back then and now.

The fact that 200,000 is a larger number than 100,000 is indeed a significant thing; it shows the gains that a company can make from increasing their scale if they are able to bring down costs enough to do so. But to  claim that this is evidence of the commercial potential of the long tail is flat out wrong. We’re still talking about a highly skewed power law distribution–in fact, an even more skewed power law distribution, as we used to speak of 20% of books accounting for 80% of the revenue, and here we are talking about 4% of the books accounting for something on the order of 99% of the revenue.

This argument appears several times throughout the book, in several forms. At one point he talks about how the scaling up of choices makes the top 100 inherently less significant. Which is true, but it does not make the head of the tail any less significant; it just means that there are a larger quantity of works within that head.

Second, this bit about “if only we can get over the economics of scarcity.” Anderson argues, repeatedly, that mass markets and big blockbusters are an artifact of a society built on scarcity, and the long tail is a creation of the new economics of abundance. This is wrong to its core.

As I argue in my first piece at The Umlaut, we have been expanding the long tail while increasing the head of the tail since the very beginning of the Industrial Revolution. Scale in the upward direction fuels scale in the outward direction. Consider Kevin Kelly’s theory of 1,000 true fans, the paradigm of the long tail success.

Assume conservatively that your True Fans will each spend one day’s wages per year in support of what you do. That “one-day-wage” is an average, because of course your truest fans will spend a lot more than that.  Let’s peg that per diem each True Fan spends at $100 per year. If you have 1,000 fans that sums up to $100,000 per year, which minus some modest expenses, is a living for most folks.

Now ask yourself: how do we get to a world where someone can make a living by having 1,000 true fans, or fewer? Or 1,000 more modest fans, or fewer?

One way we get to that world is through falling costs. If we assume a fixed amount that some group of fans is willing to pay for your stuff, then progress is achieved by lowering the cost of producing your stuff.

Another way is for everyone to get wealthier, and thus be able to be more effective patrons of niche creators. If I make twice as much this year as I did last year, then I can afford to spend a lot more above and beyond my costs of living.

Another conceivable way is sort of a combination of the first two–falling costs for the patrons. If I make as much in nominal terms as I did last year, but my costs of living fall by half, then it is effectively the same as though I had doubled my income.

Put all three of these trends together and you have perfectly described the state of material progress since the onset of the Industrial Revolution. Huge breakthroughs in our productive capacities have translated into a greater ability to patronize niche phenomena.

Obviously the personal computer and the Internet have taken this trend and increased its scale by several orders of magnitude–especially in any specific area that can be digitized. But that doesn’t mean we’ve entered a new era of abundance. The economics are the same as they have always been. The frontier has just been pushed way, way further out.

Moreover, the blockbuster is not an artifact of scarcity. Quite the opposite. The wealthier and more interconnected we are, the taller the “short tail” can be. In my article, I mention the example of Harry Potter, which was a global hit on an unprecedented scale (this Atlantic piece estimates the franchise as a whole has generated something like $21 billion). Hits on that scale are rare, giving us the illusion at any given moment that they are a passing thing, a relic of a bygone era of mass markets. But the next Harry Potter will be much, much bigger than Harry Potter was, because the size of the global market has only grown and become more connected.

Consider Clay Shirky’s observation that skew is created when one person’s behavior increases the probability that someone else will engage in that behavior “by even a fractional amount”. His example involves the probability that a given blog will get a new reader, but it extends to just about every area of human life. And the effect he describes, but does not name, is the network effect–one additional user of Facebook increases the probability that they will gain yet another one, one additional purchaser of a Harry Potter book increases the probability that yet another person will purchase it.

And we know, from the diffusion of innovations literature, that there comes a certain point at which one additional person increases the probability by a lot more than a fractional amount. As Everett Rogers put it:

The part of the diffusion curve from about 10 percent adoption to 20 percent adoption is the heart of the diffusion process. After that point, it is often impossible to stop the further diffusion of a new idea, even if one wished to do so.

Now, if network effects are what create skew in the first place, and we are living in the most networked age in history, how plausible does Anderson’s argument seem that the head of the tail will be of decreasing significance because of new networks?

What Does He Really Think?

Part of what’s frustrating about the book is that Anderson doesn’t really make a solid claim about how big he thinks the head of the tail is going to be relative to the tail. He provides some facts that are erroneous to answering this question, such as the Amazon statistic described above. In some places he seems like he’s saying the head will be smaller:

The theory of the Long Tail can be boiled down to this: Our culture and economy are increasingly shifting away from a focus on a relatively small number of hits (mainstream products and markets) at the head of the demand curve, and moving toward a huge number of niches in the tail. In an era without the constraints of physical shelf space and other bottlenecks of distribution, narrowly targeted goods and services can be as economically attractive as mainstream fare.

The long tail is going to be “as economically attractive” as the head of the tail. That’s what he’s saying, right? If so, then he is wrong, for the reasons described above.

But maybe that isn’t what he’s saying. Consider:

This is why I’ve described the Long Tail as the death of the 80/20 Rule, even though it’s actually nothing of the sort. The real 80/20 Rule is just the acknowledgment that a Pareto distribution is at work, and some things will sell a lot better than others, which is as true in Long Tail markets as it is in traditional markets. What the Long Tail offers, however, is the encouragement to not be dominated by the Rule. Even if 20 percent of the products account for 80 percent of the revenue, that’s no reason not to carry the other 80 percent of the products. In Long Tail markets, where the carrying costs of inventory are low, the incentive is there to carry everything, regardless  of the volume of its sales. Who knows—with good search and recommendations, a bottom 80 percent product could turn into a top 20 percent product.

Here he seems to be saying that the 80/20 Rule will always remain true, but that shouldn’t stop us from realizing how important the long tail is in our lives, and how much more important it will be in the future as we get ever more diversity of choices in the relatively niche. Moreover, companies should continue to extend their long tail offers because, at any moment, one of them might suddenly jump to the head of the tail. So a Kindle book that’s only selling a handful per year may suddenly go viral and make Amazon a ton of money.

If that’s what he believes, then he is correct. But the mixture of the bad accounting of the sort in the top 100,000 books example above, statements such as the one quoted above about what “the theory of the Long Tail can be boiled down to”, and this last quote about the 80/20 rule, force me to conclude that Anderson’s thinking is simply muddled on this particular point.

Credit Where Credit is Due

Finally, if there’s one thing that I think we can all agree with Anderson on, it is that the expansion of the long tail has greatly increased the quality of our lives. Whether it’s people like Scott Sigler who has managed to make a living from his fans, or the passionate community of a small subreddit, there is an ever expanding virtual ocean of choices in the long tail today.

Chris Anderson argued that the fact that something is not a hit of the blockbuster variety does not mean it is a miss. There are some things that are much more valuable to a small group of people than they are to everyone else, thereby precluding their ability to become a blockbuster. There are also some things that might be equally appealing to the same number of people as a blockbuster, but they simply were not lucky enough to be among the few that won that particular lottery.

All of us live in both the head of the tail and the long tail, and I’m glad that Anderson convinced so many of the value of the latter.

Information, Food, and Democracy

I am a skeptic on the subject of food diets but have been a true believer in the information diet for a long time. I am pretty sure that Clay Johnson had the opposite sort of reader in mind when he wrote The Information Diet.

As I read the book, I kept thinking of things I wanted to respond to and aspects I wanted to explore. But in the spirit of restraint which the book embodies, I will hold myself to just two posts – an overview of my own information diet, and this broader response.

Information Mania

Back in 2004 I was extremely into the political blogging scene and thought that all mainstream news was utter crap. My reasoning was pretty straightforward – bloggers actually linked to things in order to provide context. They didn’t pretend to be objective; they wore their ideologies and biases on their sleeves. The distorting news cycle that mainstream media operated on, and the narrow perspective they approached everything from, drove me crazy. I excluded it from my information diet as much as possible.

I had family members and friends who I could see develop a kind of mania as they followed more traditional news outlets. They didn’t want to talk about anything but politics. They made blunt statements that were insulting to anyone who disagreed with them, under the assumption that no one did, and were genuinely surprised when it turned out this assumption was wrong.

The section of Johnson’s book where he talks about his relationship with his Uncle Warren, a Fox News faithful conservative, really touched a chord with me.

The conversation, viewed from the outside, couldn’t have been considered coherent. It was just an explosion of nonsense—two grown men shouting at each other about taxes, gun control, and healthcare, with some perfectly good turkey between us. That Thanksgiving, Uncle Warren left early, and things haven’t been the same between us since.

I myself came right to the brink of such a fallout on several occasions. Of course I realize now what Johnson has also realized; that my own behavior and my own information diet were part of the problem. I fell into an “us versus them” mentality and I selected for blogs that fed into that. Despite adamantly refusing to affiliate myself with either political party, I was still drawing a line in the sand; my people against everyone else.

It was extremely bad for me. It made me unhappy and dissatisfied, and every so often I would become downright unpleasant company.

I was right about mainstream news, but was unwilling to recognize that the problem was not the medium but the type of content. Since I considered myself a part of the new media crowd, I didn’t give new media as critical an eye as I did traditional media. The only exception was the critical eye I turned to those blogs of a different point of view.

I’m still a techno-optimist and I still think that most news is crap, but I have come to realize that there is a lot more crap on the Internet than off of it. This is because there is a lot more of everything on the Internet, good and bad. The digital age gives the producers of crap some new, powerful tools that they didn’t have before, as Johnson describes:

Even more than television, Fox routinely tweaks the news on the Web to make the news more palatable to its audience. Even when it takes content from other sources like the Associated Press and puts it on its website, the organization tweaks the headlines to make them more attractive to its conservative audience. The AP’s story “Economic Worries Pose New Snags for Obama” turned into “Obama Has a Big Problem with White Women.” “Obama to Talk Economy, Not Politics, in Iowa” turned into “White House Insists Obama’s Iowa Stop for Economy, Not 2012.” And “Malaysia Police Slammed for Cattle-Branding Women” turned into “Malaysian Muslims Cattle-Brand Prostitutes.”

Fox isn’t about advancing a conservative agenda. For its parent, News Corporation, it’s about the dollars. Fox changes these headlines on the Web not because it has an agenda, but because people click on them more, meaning that more advertisements can be shown, and more money can be made.

So I have, for some time now, agreed with the general proposition that the information we consume has an effect on us on our emotional and intellectual state of mind. This is an idea that is old and familiar to me, and it is a crucial part of the story told in The Information Diet.

But it isn’t the whole story.

Food Morality

Something else that is common among certain members of my family and friends is the propensity to moralize on the subject of food.

A lot of the time, this centers on the “obesity crisis” or “obesity epidemic”; something that is apparently a very important public health issue. An uncle of mine once assured me that it was only a matter of time before national life expectancy began to fall as a result of this problem.

The food moralist’s story is a key piece of the story that Johnson wants to tell in The Information Diet. Consider the following:

Today, 13.5 million people die each year of heart disease and stroke, and 4 million from diabetes-related complications–far more than die in automobile accidents. heart disease is now our number one killer, and it takes more people to the grave in the United States in five years than all our war-related deaths combined. Instead of dying from the cold of winter, we find death in cholesterol.

The way this story is framed is at odds with my understanding of the material progress that has been made in developed nations over the last two hundred years. But I didn’t want to just dismiss it out of hand, so I went to see if the known facts were what I had thought that they were.

The following are the facts, as I understand them, with links to primary sources.

First, my uncle’s comment aside, life expectancy has been rising consistently for centuries. It’s been rising more slowly in the last few decades than it was, say, right after the discovery of penicillin. Nevertheless, it has continued to increase. I understand that the past is not prophecy; it could start to decline. But it hasn’t so far.

But what about the specific risk factors of being obese? What about heart disease and stroke? After all, as Johnson points out, “heart disease is now our number one killer.”

Just because something is the biggest relative cause of death does not mean that it is a bigger problem than it used to be.

According to the CDC, “Overall, death rates for heart disease and stroke have decreased in the United States in recent decades.” They hasten to add that “rates for incidence and death continue to be high”, and to talk about how it is higher among some socioeconomic groups than others. But they don’t include a breakdown of whether those higher rates have increased or declined – a number I would be very interested in seeing.

According to the American Heart Association, “From 1997 to 2007, the death rate from [cardiovascular disease] declined 27.8%” and “the stroke death rate fell 44.8%, and the actual number of stroke deaths declined 14.7%.” Not only is the stroke death rate down, the actual number of deaths is down despite the larger population. This is not the rate at which people who contracted these diseases died from it – this is the number of people that died of these diseases per 100,000 people in the overall population.

This 1997-2007 period is squarely in the middle of the 1990-2010 period that Johnson cites as one of enormous growth in the obesity rate. People were getting fatter but the things that they are at higher risk for was killing them less often. This is hardly finding “death in cholesterol”.

Now, these are just the death rates–it could be that the incidence rate of heart disease and strokes has increased, but medical science has advanced so far that it’s actually overcompensated and the number of people per 100,000 in the population that actually died of them decreased. And I’m not saying that it’s a good idea to be overweight, or that you can eat nothing but chocolate for three meals a day, seven days a week. Obviously dietary choices have health consequences; a friend of mine knew someone in college who ate nothing but ramen for a month and actually ended up getting scurvy.

I just don’t see the smoking gun here. This is no existential threat, no “crisis”; this is a lot of people eating in a less healthy manner than a physician would probably recommend. But they are still, on average, living longer. In particular, the things that overweight people are at risk for are killing them less often.

I believe that what we eat is a personal and quality of life issue, not a moral one. Some people may have less energy and feel bad about themselves because of the kind of food they eat regularly. On the other hand, some people may be happy eating food that they enjoy but being overweight to some degree. There are trade-offs to be made, and everyone must weight those trade-offs for themselves.

This disagreement over whether food is a moral issue or a personal one is paralleled in the conversation about information.

Saving Democracy

Johnson does not just want you to fix your information diet for the sake of self-improvement. He wants you to do it because he thinks it will be good for democracy.

The role that information plays in influencing politics is complicated and difficult to measure. It’s something that my father thinks long and hard about over at The Fifth Wave.

Johnson appears to hold to what I would describe as the traditional progressive hypothesis: the more informed the governed, the better the quality the governance in a democracy will be.

once you lose the fluff and start really seeing what’s going on, new priorities arise that require new tactics to accomplish.

He recognizes that there isn’t a straightforward relationship between the amount of information available and policy outcomes. For instance, he spends a whole section taking down the notion that mere “transparency” is actually effective at holding policymakers accountable.

Like the calorie counts from food, transparency is ineffective at arming the masses unless there’s a strong will in the public to arm itself with the knowledge of how this information affects us, and how to effectively read the metaphorical labels. People will be no less obese—and no less ignorant—unless they have the will to consume less of the stuff that’s bad for them, and more of the stuff that’s good for them. While transparency can help the problem, it alone cannot fix it.

Admitting that mere transparency isn’t enough to achieve mass enlightenment does not mean that Johnson thinks such enlightenment is impossible, however.

He then offers a laundry list of how you can make a difference, ranging from the kinds of problems you should focus on:

My plea to you is to start sweating the small stuff at the expense of some of the big stuff. Washington isn’t the land of vast, radical changes, it’s a battleship waiting to be nudged in the right direction.

Nudges such as:

If you’re worried about federal spending and the budget, don’t concern yourself over the debt-ceiling debate. Work to change procurement laws so that government can get access to the same things the private sector has without paying an arm and a leg. We spend so much time figuring out what programs to spend money on, comparing their priorities to one another, and blanket cutting them when they’re deemed too luxurious. It’s the equivalent of trying to lose weight by cutting off your legs. Optimizing how government spends its money is at least as important as figuring out what our money gets spent on, and there are real, pragmatic solutions to getting there.

There are also tools available to us, and tools yet to be made, that can help us do the job.

We can also improve our government without waiting on government to act. Organizations like PopVox.com, for instance, make it easier for people to translate what they want their representative to do into the language our representatives speak. There’s a whole world of technology out there waiting to be used to help members listen to their constituents, and it’s likely—now that much of our discussions about politics are public—that we don’t need government to act: we can build tools that listen to what people are already saying, make that information public, and question our elected officials when they’re voting against their constituencies.

I’m not denying the value in what he’s saying, I just really wonder how much exhortations like these are going to make a difference.

I don’t pretend to any special knowledge on the relationship between information and politics, but it seems to me that when you write a book encouraging people to have a nice, balanced, reasonable information diet, it encourages people to be nice, balanced, and reasonable. In my admittedly limited experience, it is rarely the nice, balanced, and reasonable people that move history.

I keep going back to the circumstances where policy or norms moved in a direction that I, personally, consider to be good. For instance, when the web rose up against SOPA and PIPA and managed to get those bills tabled. In my opinion, it wasn’t the well informed, reasonable people that made that happen. It was Wikipedia and other major websites going dark and essentially sending the message “STOP SOPA OR NO MORE WIKIPEDIA/LOLCATS FOR YOU”, lighting a fire under the typical web user to get pissed off in their representatives’ direction.

Or a much more important example, the civil rights movement. Martin Luther King and the student activists of the 1960’s were not just nice and reasonable people, they were extremely shrewd about the media of their day and the rhetoric that they used. They picked their battles very carefully to have the maximum emotional impact.

That is what I thought of when I read the following line in The Information Diet:

As much as I’d like to use the political skills I’ve learned in the past 10 years to drive a traditional campaign, doing so would go against the principles of the book.

Call me cynical, but I tend to think that if the problems that Johnson thinks are solvable do get solved, it will be because some very effective propagandists and rhetoriticians get behind it. It won’t be because people are suddenly more reasonable and informed, and decide en masse to get behind the solving of practical problems.

That’s just my instinct. It doesn’t help that I went to a grad school that distinguished itself in the field of Public Choice theory, which tends to make you skeptical of the notion that a more informed populace translates into better policy outcomes.

I should also say that I’m skeptical of the hypothesis that things are so much worse and so much more polarized than they were in the past. I think the fact that we all share this big information space called the Internet means we’re exposed to the loud and angry among us much more than we used to be. I also think that technological change and an increased overall wealth now allows us to support the Fox News’ and MSNBC’s–the Krispy Kreme donuts of information intake–rather than the bland mass products of old like CBS News. But I don’t really think those bland mass products were any better for us.

A Book Worth Reading

So I don’t really think a good information diet is going to save democracy, but I do think that there are big benefits to be had by taking the time to scrutinize your information diet. Johnson made me think not just about the content I consume, but the amount of time I spent obsessively checking for updates.

The Information Diet is, for me personally, exactly the kind of meal I need to have more often in my own information diet. It challenged my perspective on a number of margins but was reasonable and inoffensive while doing so. More than anything, it made me think. It also introduced me to interesting tools like EveryBlock.

I really can’t recommend this book enough. It’s only 160 pages long and very well written. I got more out of reading it than I have from books three times its length.

Once you’ve read it, I recommend Darrell Huff’s classic How to Lie with Statistics as a good companion piece. It’s even shorter than The Information Diet and is an excellent guide to sorting the sense from the nonsense.