Random thoughts on being random.

The story is told that Bennett Cerf, who founded the Random House publishing company, got the name for his company from the conversation he had with his partners about what he was going to do: just publish some books on the side at random. The rest, as they say, is history. Random House is now owned by Bertelsmann, which I'm sure never even crossed Bennett Cerf's mind as something that might happen. That acquisition seems almost a random event.

Randomness is everywhere. Open the newspaper, and chances are that a few of the articles describe some study or other: some sort of food causes cancer, lots of people don't like the President, fewer people are going to the dentist. Sometimes these studies claim to prove other studies wrong, sometimes they are new "discoveries," and sometimes they just seem, well, random. But the key to all these studies and surveys and experiments is that they in fact are random. They have to be. Randomness is built into the process. If the objects being studied are not chosen randomly, the study isn't worth anything. At least that is the theory: choosing subjects at random ensures that there aren't any patterns in the process of choosing the subjects that can affect the results.

The most famous polling fiasco was Literary Digest magazine's poll for the 1936 presidential election. Based on two million responses, the Literary Digest projected a landslide for Alf Landon. The problem is that the people who responded to the survey weren't randomly chosen. The magazine had used driver's license records and telephone numbers to identify people to whom to send the survey, but back then many people didn't have driver's licenses or telephones, mainly because they could not afford cars or phones or lived in places that were too remote to have driveable roads or telephone service. Not only that, but the only people who responded were those who chose to respond. That means the survey was based on responses by a self-selected group taken from a skewed sample. The sample size was huge but it wasn't representative of anything. No wonder the results were worthless.

Use randomly chosen study subjects, though, and magically the results become more reliable. Back in college in my statistics course I learned that the smallest reliable sample size is thirty, and that the margin of error gets smaller as the sample size grows above that. But large sample size is useful only if the sample is chosen randomly. Introduce any method to the choosing other than total randomness and you skew the results.

What is remarkable about this process is the use of chaos to produce order. There is information out there about tendencies and preferences. Lots of information. How do we get at that information? How do we find out whether cholesterol tends to cause heart attacks? How do we find out what most people think about our elected representatives? The answers are out there, but we need to extract them from the huge mass of other information that is also out there. The way we get the answers -- the way we impose order on the chaos -- is by using chaos, randomness. If we use order, a pattern, we can't get a reliable answer, but if we use no pattern at all, we can.

The uses of disorder and chaos are one of the great paradoxes of life. Evolution itself, the very basis of our existence as hominids, is based on random mutations that turned out to have had adaptive uses. Take away the random mutations and we might never have grown our outsized brains or even developed chins. We might even still have lots of body hair, copulate only when women are in heat, and brachiate more readily than walk.

Randomness sometimes explains a lot: as the saying goes, give a monkey enough time at the keyboard and eventually he'll produce War and Peace. Or Shakespeare. The idea, of course, is that enough randomness over enough time can yield just about any wondrous thing.

Even our everyday speech recognizes the effectiveness of lack of order. Think of the popular injunction about how important it is to do "random acts of kindness." Why random? So that anyone at all could benefit, no matter how unconnected to you. Randomness spreads the love around.

So that raises an interesting question. If randomness, disorder and chaos are so useful, why can't people handle them better? Think of it: isn't a pattern the first thing you look for in life? Babies learn about the world by observing patterns. They learn that objects fall down. They learn that if they cry Mommy comes running. Adults learn from patterns, too. We usually construct our lives based on those patterns. We know it's safe to cross the street when the light is green, because the drivers who are heading in a perpendicular direction to us stop at what for them is a red light. We know it's usually colder in January than in June. The list of examples goes on.

But there's an awful lot of chaos and disorder out there, and much of the trouble in life comes from things that don't follow the patterns we expect and break out of the routines we have constructed. New Orleans would still be standing if the old patterns about weather had held. The Soviet Union never would have imploded if prior patterns had held in 1989. We would still be using carbon paper and typewriters if previous technological patterns had not been broken by new ideas and technology. Economic projections are based on prior patterns, but can get thrown into a cocked hat by an unforeseen event. When life throws us these curveballs we have to scramble to recover and deal with the new situation.

My question is, why? If chaos is so pervasive, if patterns are so constantly disrupted, why don't we have the mental architecture to assimilate disorder? Especially since order can be produced by chaos, why can't we deal with the chaos directly, instead of trying to impose order on it? It's almost like we are vassals of the need to see patterns. When the chaos strikes and the patterns are disrupted, first we identify the variation in the pattern and then we try to figure out whether we now have a new pattern, whether we should try to go back to the old one and whether the chaos can be tamed. But we always view the chaos as a departure from the norm, rather than part of the norm. Why? Shouldn't we be able to assimilate the chaos and manipulate it directly, instead of first placing a pattern-based template over it?

If chaos can produce order, maybe chaos isn't such a bad thing. We should try to use it creatively.